Instant messaging with emotion-embedded vectorized handwritings on mobile devices

Syu et al. EURASIP Journal on Image and Video Processing (2017) 2017:23 DOI 10.1186/s13640-017-0173-6 EURASIP Journal on Image and Video Processing R ES EA R CH Open Access Instant messaging with emotion-embedded vectorized handwritings on mobile devices Nai-Sheng Syu1 , Jia-Wei Kuo2 , Chih-Yuan Yao3 , Shan-Hsiang Shen3 and Yu-Chi Lai3* Abstract Generally, handwriting can reflect writers’ personality, thoughts, and emotions, i.e., handwriting can deliver emotionand sincereness-embedded messages. However, texting messages and notes such as emails and instant messages replace handwriting letters and notes in communication due to the popularity and availability of mobile devices and personal computers. Furthermore, the commonly used input methods and devices also limit handwriting messaging. For example, limited mobile screen sizes make writing multiple Chinese characters difficult. As a result, this work aims at designing and developing a handwriting messaging system based on our handwriting characteristic exploration and texting-handwriting difference discovery. We first discover Chinese texting issues for mobile devices and emotion-delivered effectiveness of handwriting with pilot studies. Then, our emotion-embedded handwriting messaging framework is implemented to record writing strokes on the entire touch screen, vectorize them as Bézier curves, and send them as instant messages. Vectorizing strokes can preserve personal and emotional handwriting features with compact networking traffic without deteriorating the convenience of instant messaging. Finally, we conduct user studies to verify that our handwriting messaging system is preferred while intending to deliver contents with sincerity and emotion. Keywords: Emotional handwriting, Effective emotion delivery, Emotional texting, Vectorized handwriting 1 Introduction With the advance of technology, computer-based personal messages such as emails and instant messages become popular and replace handwriting letters and notes in communication. Mobile devices make them even more universally popular. Users generally choose a preferred systematic style and font for texting, and this makes messages lack of personality, emotion, and sincerity. Because different users may have different handwriting, and even the same person may have different handwriting under different emotions, it is easier to deliver emotion, feeling, and sincerity. Therefore, this work aims at adding handwriting characteristics into instant messaging for more personality and emotion. *Correspondence: Yu-Chi Lai is funded by MOST 104-2221-E-011-029-MY3, 103-2221-E-011-114-MY2, and 103-2218-E-011-014, Taiwan 3 National Taiwan University of Science and Technology, Taipei, Republic of China Full list of author information is available at the end of the article Before further describing and discussing the focus and technical details of this article, we define the terms used in this article for clarity. Traditionally, handwriting is used to describe the process of writing with a pen or pencil in the hand, and this work uses it to denote the act of composing digital messages with an electronic device while maintaining writer’s personal writing characteristics including stroke shapes and orders. Handwriting messages and notes are the products of handwriting. Similarly, we use texting to describe the process of composing digital messages encoding with the format for personal computers and mobile devices, typically consisting of alphabetic and numeric characters. Currently, there are three popular input methods including typing, speech recognition [1–4], and stroke recognition [5–9]. We give the details of each input method in Section 3. Texting messages and notes are the products of texting. This work uses messaging/instant messaging to describe the act of sending and receiving electronic messages between two or more users with stationary and mobile devices. © The Author(s). 2017 Open Access This article is distributed under the terms of the Creative Commons Attribution 4.0 International License (http://creativecommons.org/licenses/by/4.0/), which permits unrestricted use, distribution, and reproduction in any medium, provided you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons license, and indicate if changes were made. Syu et al. EURASIP Journal on Image and Video Processing (2017) 2017:23 Therefore, we use handwriting texting messaging to denote the act of messaging under the contents composed by handwriting/texting respectively. There are a variety of issues while texting with those commonly used input methods including typing with a keyboard, stroke recognition, and speech recognition. Although typing with a keyboard is still the prevalent method, it requires practicing to get familiar with the setting for gliding typing. For example, there are multiple homographs for a Chinese sound; multiple selections may pop up when using phonetic input. Typing is hard for entering special symbols. Stroke recognition [1–4] based on stroke orders and character structures is an alternative, but recognition rate is an issue. It is even worse on mobile phones due to the fat finger issues. Another popular method is speech recognition [5–9], but recognition rate is problematic, and some environments are not proper for certain environments and situations such as a noisy station or a meeting occasion. We conduct a set of pilot studies to verify and assert these commonly observed issues. Furthermore, the results of these studies also let us design to overcome these limitations by directly collecting the hand-written characters/symbols based on written tracks on the touch screen for messaging due to the following reasons: 1. Handwriting can more easily deliver emotion and personality. 2. Handwriting can directly express their emotion with drawings with less constraints in remembering the vocabularies and symbols existing in other input methods. 3. Handwriting can also achieve availability, portability, convenience, and instantaneity which are important requirement for instant messaging. After implementation, we conduct a set of user studies to verify that our system can let users prefer handwriting over other texting input methods in some scenarios due to easiness in expressing personality and emotions and arbitrary symbol drawing with few constraints. Moreover, our proposed input can free users from frustration of writing recognition to get better experiences and make instant messaging more intuitive and friendly. This paper makes the following contributions: First, we identify an interesting and important input method for instant messaging on mobile devices: an intuitive and emotion-embedded handwriting input framework. Second, the core of our method is to record and send user’s handwriting messages with vectorized strokes using Bézier curves for preservation of personal and emotional handwriting features with co (...truncated)