Eye-blink detection system for human–computer interaction

Aleksandra Krolak 0 Pawe Strumio 0 0 A. Krolak (&) P. Strumio Institute of Electronics, Technical University of Lodz , Wolczanska 211/215, 90-924 Lodz, Poland A vision-based human-computer interface is presented in the paper. The interface detects voluntary eyeblinks and interprets them as control commands. The employed image processing methods include Haar-like features for automatic face detection, and template matching based eye tracking and eye-blink detection. Interface performance was tested by 49 users (of which 12 were with physical disabilities). Test results indicate interface usefulness in offering an alternative mean of communication with computers. The users entered English and Polish text (with average time of less than 12s per character) and were able to browse the Internet. The interface is based on a notebook equipped with a typical web camera and requires no extra light sources. The interface application is available on-line as open-source software. Human-Computer Interface (HCI) can be described as the point of communication between the human user and a computer. Commonly used input devices include the following: keyboard, computer mouse, trackball, touchpad - and a touch-screen. All these devices require manual control and cannot be used by persons impaired in movement capacity. Therefore, there is a need for developing alternative methods of communication between human and computer that would be suitable for the persons with motor impairments and would give them the opportunity to become a part of the Information Society. In recent years, the development of alternative humancomputer interfaces is attracting attention of researchers all over the world. Alternative means of interacting for persons who cannot speak or use their limbs (cases of hemiparesis, ALS, quadriplegia) are their only way of communication with the world and to obtain access to education or entertainment. A user friendly humancomputer interface for severely movement impaired persons should fulfill several conditions: first of all, it should be non-contact and avoid specialized equipment, it should feature real-time performance, and it should run on a consumer-grade computer. In this paper, a vision-based system for detection of voluntary eye-blinks is presented, together with its implementation as a HumanComputer Interface for people with disabilities. The system, capable of processing a sequence of face images of small resolution (320 9 240 pixels) with the speed of approximately 30 fps, is built from off-theshelf components: a consumer-grade PC or a laptop and a medium quality webcam. The proposed algorithm allows for eye-blink detection, estimation of the eye-blink duration and interpretation of a sequence of blinks in real time to control a non-intrusive humancomputer interface. The detected eye-blinks are classified as short blinks (shorter than 200 ms) or long blinks (longer than 200 ms). Separate short eye-blinks are assumed to be spontaneous and are not included in the designed eye-blink code. Section 2 of the paper includes an overview of earlier studies on the interfaces for motor impaired persons. The proposed eye-blink detection algorithm is described in Sect. 3. Section 4 presents the eye-blink controlled human computer interface based on the proposed algorithm. Research results are discussed in Sect. 5 and the conclusion is given in Sect. 6. 2 Previous work For severely paralyzed persons who retain control of the extraocular muscles, two main groups of humancomputer interfaces are most suitable: braincomputer interfaces (BCI) and systems controlled by gaze [1] or eye-blinks. A braincomputer interface is a system that allows controlling computer applications by measuring and interpreting electrical brain activity. No muscle movements are required. Such interfaces enable to operate virtual keyboards [2], manage environmental control systems, use text editors, web browsers or make physical movements [3]. Braincomputer interfaces hold great promise for people with severe physical impairments; however, their main drawbacks are intrusiveness and need for special EEG recording hardware. Gaze controlled and eye-blink-controlled user interfaces belong to the second group of systems suitable for the people who cannot speak or use their hands to communicate. Most of the existing methods for gaze communication are intrusive or use specialized hardware, such as infrared (IR) illumination devices [4] or electrooculographs (EOG) [5]. Such systems use two kinds of input signals: scanpath (line of gaze determined by fixations of the eyes) or eyeblinks. The eye-blink-controlled systems distinguish between voluntary and involuntary blinks and interpret single voluntary blinks or their sequences. Specific mouth moves can also be included as an additional modality. Particular eye-blink patterns have the specific keyboard or mouse commands assigned, e.g., a single long blink is associated with the TAB action, while a double short blink is a mouse click [29]. Such strategies can be used as controls for simple games or for operating programs for spelling words. The vision-based eye-blink detection methods can be classified into two groups, active and passive. Active eyeblink detection techniques require special illumination to take advantage of the retro-reflective property of the eye. The light falling on the eye is reflected from the retina. The reflected beam is very narrow, since it comes through the pupil and it points directly toward the source of the light. When the light source is located on the focal axis of the camera or very close to it, the reflected beam is visible on the recorded image as the bright pupil effect (Fig. 1). The bright pupil phenomenon can be observed in the flash photography as the red eye effect. Fig. 1 Eye image in IR spectrum: a eye in natural lighting conditions, b bright pupil effect after IR illumination of the eye An example of the gaze-communication device taking advantage of IR illumination is Visionboard system [4]. The infrared diodes located in the corners of the monitor allow for the detection and tracking of the users eyes employing the bright pupil effect. The system replaces the mouse and the keyboard of a standard computer and provides access to many applications, such as writing messages, drawing, remote control, Internet browsers or electronic mail. However, the majority of the users were not fully satisfied with this solution and suggested improvements. A more efficient system was described in [9]. It uses two webcamsone for pupil tracking and second for estimating head position relative to the screen. Infrared markers placed on the monitor enable accurate gaze tracking. The developed system can replace the computer mouse or keyboard for persons with motor impairments. The active approach to eye and eye-blink detection gives very accurate results, and the method is robust [8]. The advantages of the IR-based eye-controlled human (...truncated)