Recognition and Classification of Sign Language for Spanish

Recognition and Classification of Sign Language for Spanish Griselda Saldaña González1 , Jorge Cerezo Sánchez1 , Mario Mauricio Bustillo Dı́az2 , Apolonio Ata Pérez2 1 Universidad Tecnológica de Puebla, Ingenierı́a en Tecnologı́as para la automatización, Puebla, Mexico 2 Benemérita Universidad Autónoma de Puebla, Facultad de Ciencias de la Computación, Puebla, Mexico {griselda.saldana, jorge.cerezo}@utpuebla.edu.mx, , Abstract. In this paper it is presented a computational system for recognition and classification of letters of the sign language in Spanish, designed for helping deaf-mute people to communicate with other persons. A low-cost glove that captures the hand movements has been constructed. This one contains an accelerometer for each finger which allows detecting its position by using an acquisition data board. Sensor information is sent wirelessly to a computer having a software interface, developed in LabVIEW, in which the symbols dataset is generated. For the automatic recognition of letters we have applied a statistical treatment to the dataset obtaining accuracy greater than 96% independently of the user. Keywords. Signs language, machine learning, glove. 1 Introduction Recognition of signs has been the focus of several research areas such as human-computer interaction, virtual reality, tele-manipulation and images processing. Another area of application is sign language interpretation [1]. Among the types of gestures, sign language is one of the most structured; usually each gesture is associated to a predefined meaning. However, the application of strong context rules and grammar makes sign language more difficult to recognize [22]. According to sensing technology used to capture gestures, there are two main approaches for sign recognition. One based on vision techniques [16], in which hand movement is followed and the corresponding sign is interpreted [23, 18] and another based on gloves [15], with sensors that capture the movement and rotation of hands and fingers [9]. Other methods include Leap Motion [11], or Kinect sensors [17]. Corresponding to the approach based on vision; in [20] a method to convert the Indian Sign Language (ISL), hand gestures into appropriate text message is presented. The hand gestures are captured through a webcam and the corresponding frames are segmented considering features such as number of fingers and the angle between them. Trigueiros et al. [25], used vision based technique for recognition of Portuguese language. For their implementation, hand gesture was captured in real time. SVM algorithm is used for classification purpose. In this system vowels are recognized with accuracy of 99.4% and consonants are recognized with 99.6% accuracy. In [3], a real-time method for hand gesture recognition is presented. The hand region is extracted from the background, then the palm and fingers are segmented to detect and recognize the fingers. A rule classifier is applied to predict the labels of hand gestures. Computer vision based techniques have the potential to provide more natural and non-contact solutions, and are based on the way human beings perceive information about their surroundings [21]. The main drawback is in the acquisition process due to many environmental apprehensions such as the place of the camera, background condition and Computación y Sistemas, Vol. 22, No. 1, 2018, pp. 271–277 ISSN 1405-5546 doi: 10.13053/CyS-22-1-2780 272 Griselda Saldaña González, Jorge Cerezo Sánchez, Mario Mauricio Bustillo Díaz, Apolonio Ata Pérez lightning sensitivity [14], in addition accuracy and processing speed are challenging. Leap Motion controller is a small USB device that using monochromatic IR cameras and infrared LEDs, observes a roughly hemispherical area, to a distance of about 1 meter. The LEDs generate pattern-less IR light and the cameras generate almost 200 frames per second [26]. P. Karthick et al. [10] used a model that transform Indian sign language into text using a leap controller. The Leap device detects the data like point, wave, reach, grab which is generated by a leap motion controller. Combination of DTW and IS algorithm are used for conversion of hand gesture into text. Neural network was used for training the data. In [6] a leap motion controller is used for recognition of Australian sign language. Leap motion controller senses the hand movement and convert that hand movement into computer commands. Artificial neural network is used for training symbols. The disadvantage of that system was low accuracy and fidelity. With the emergence of RGB-D (color images and depth maps synchronized) and capture devices, using mainly the Microsoft Kinect sensor; the gesture recognition field had a great push forward [12]. In [5] a Microsoft kinect was used to recognize American Sign Language (ASL). Depth camera is Kinect sensor used to detect ASL alphabet. Distance adaptive scheme was used for feature extraction. Support vector machine and RF classifier algorithm used for classification purpose. Training of data was done using neural network. The accuracy of the system was 90%. In [2] a 3D trajectory description of one sign language word is used and matched it against a gallery of trajectories. Another work presented in [7] used an RGB-D image from the Microsoft Kinect sensor to recognize the letters of the manual alphabet, known as fingerspelling. These works used data from a point cloud and required further processing for hand detection before actually detecting gestures. The Leap Motion skip this step, because already handles the detection by itself. Recognition based on sensors such as accelerometers and gyroscopes offer the following advantages: a) because movement sensors are not affected by the surrounding, recognition is Computación y Sistemas, Vol. 22, No. 1, 2018, pp. 271–277 ISSN 1405-5546 doi: 10.13053/CyS-22-1-2780 more adequate than recognition based in vision in complex surroundings b) they are joined to a user, this allows a bigger coverage, and c) the signs can be acquired wirelessly [13]. Gloves have been successfully used for the recognition of signs in previous works [4, 24], in [1] a system for the recognition of the 23 letters of the Vietnamese language is presented; this system uses a glove with accelerometers MEMS, whose data is transformed to relative angles between the fingers and the hand palm. For the recognition of the letters, it uses a classification system based in fuzzy logic. In [27] a glove based in accelerometers and myoelectric sensors is reported, its elements allow it to automatically detect the initial and final point of two significative segments of the symbols by the intensity of the myoelectric sensors. To obtain the final result, it uses decision trees and hidden models of Markov. The functionality of the system is shown by the classification of the 72 symbols of Chinese sign language. [8] presents a framework for Sign Language Gesture recognition using an a (...truncated)