Comparative analysis of affective and physiological responses to emotional movies
Kim et al. Hum. Cent. Comput. Inf. Sci.
Comparative analysis of affective and physiological responses to emotional movies
Kyoung Shin Park
In this research, we investigated on user's affective and physiological responses to the emotional movies. The emotional movies provide continuous feedback such as changes in saturation, brightness and contrast of the movie in response to users' emotional states. In the user study, the subjects watched the emotional movies, edited by fearful and joyful scenes, and presented in 2D and 3D formats. The subject's self-reported emotion responses and their physiological signals were analyzed. The results highlight the importance of scenes (such as, the color, tone, and brightness of the scene) than 2D or 3D format in understanding the impact of users' emotional responses. More physiological emotional changes, positive user responses and higher correlation rate between physiological and subjective responses were evident for joyful scene than fearful scene when presented in 3D format.
Emotion elicitation; Emotional movies; Affective response; Emotionally intelligent contents
The three-dimensional movies are popular among today’s generation. Following the
success of the blockbuster movie Avatar, interest in 3D movies has increased significantly,
leading to the production of more 3D movies. 3D televisions, 3D broadcasting and 3D
Blu-ray movies also hit the market. More recently, virtual reality becomes a popular
trend for consumers who want even more immersive 3D experiences. A few months ago
Paramount Pictures launches the first virtual reality movie theater.
Along with the rising popularity in 3D technologies, the biggest concern of creating 3D
contents is to offer deeper and more meaningful experiences. Many works have shown
that 3D images help users experience excitement, a feeling of presence, and enjoyment,
and even performance advantages for depth-related tasks [
]. Despite the growing
interest in 3D contents and technologies, few are concerned about viewers’ perceptual and
emotional responses to 3D stimuli.
Much 3D related research is focused on technical factors, such as the fidelity of
stereoscopic depth and reduction of visual discomforts while the viewer is watching 3D
]. For example, subjective evaluation was used to compare visual fatigue induced by
© The Author(s) 2018. This article is distributed under the terms of the Creative Commons Attribution 4.0 International License
(http://creativecommons.org/licenses/by/4.0/), which permits unrestricted use, distribution, and reproduction in any medium,
provided you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons license, and
indicate if changes were made.
2D and 3D high-definition television (HDTV) [
], but subjective evaluations often show
considerable variation between individuals.
On the other hand, physiological measurements provide a more objective evaluation
for studying visual fatigue induced by a 3D display [
]. For example, the study on
influence of 3D TV based on physiological signals measured the viewers’ emotional states [
]. Results showed a correlation between the effect of physiological responses and
emotional states. This indicates that physiological responses can be potentially ample
indicators of emotionally engaging 3D video.
Human emotion is a complex phenomenon with both affective and physiological
components. The affective component refers to the users’ subjective feeling while the
physiological component refers to the physiological responses to user emotion. In affective
computing, many researchers strive to understand how human emotions are evoked in
response to multimedia contents. Some studies have shown that specific audio-visual
stimuli affect changes in user emotion [
]. More attention has been devoted to
measuring physiological responses to 2D or 3D movie . However, these are rather passive
and one-sided, and the contents do not reflect users’ affective states.
In this research, we investigated on the effects of the emotional movie. The emotional
movie means the movie that dynamically adjusts its visual appearances, such as
saturation, brightness and contrast of the scene according to users’ emotional states in
realtime. This approach is essential for creating more engaging and affective environments.
Previous research has revealed that such emotional movies had significantly more
influence on the subject’s emotional states than the original movies [
Despite the need for understanding the interactive nature of user emotions elicited by
the emotional movie, little is known about the impact of 3D emotional movie and the
role that different colors or tones of light play in the scene. This is challenging because
emotional response is complex, and arises not just from the video itself, but from the
context of the video, such as the storyline, the colors or tones of the scene, and even
from the 3D effects.
This research aims to provide the basis for objective and subjective measurements of
users’ emotional responses to emotional movie. In a comparative user study, the
participants viewed the emotional movie, edited by fearful and joyful scenes and shown in
2D and 3D format. While viewing the emotional movie, the degree of user attention,
emotional sensitivity, memory, interest and fatigue were measured using objective
physiological signals and subjective responses.
The overall paper is structured into seven sections. “Background” section reviews the
related background information about studies on emotional contents. “System design
and implementation” section presents the system design and implementation of the
emotional movie. “Methods” section presents the experimental method for
assessment of affective and physiological responses in regards to 2D and 3D emotional movie.
“Results” and “Discussion” sections describe the results and discussions. The conclusion
and directions for future research are discussed in “Conclusion” section.
There is a great deal of research on how color influences emotional changes in users. For
example, the international affective picture system (IAPS) research extracted emotion
images from pictures, showed them to users and then measured their emotional changes
]. There is a study that adjusted colors to fit an individual’s environment in order to
find the color that reduces stress the most and creates the most calming environment for
the individual [
Still other research looked at how the color of light affects cognitive functions, such as
attentiveness, memory and emotional change. For example, there is a study that
evaluated the impact of color hue, brightness and saturation of e-commerce websites on
customer emotions and trust. Results showed that more saturated and brighter colors can
motivate consumers to have positive reactions to the site [
On the other hand, there have been studies conducted to compare 2D and 3D
technologies. For example, there has been a study looking at whether changes in presence (a
sense of being there) exist for different genres of 3D TV programs [
] and a study
investigating the effect of genre on 3D movies [
]. It also studied whether 3D images offer
the same amount of user satisfaction [
], how 3D images influence the users’ biological
signals and awakening [
], and how 3D games affect the level of user engagement as
compared to 2D games [
These works have studied emotional changes by adjusting the original content before
or after showing them to users and studying whether there is a difference in sense of
realism, user satisfaction, and level of engagement between 2D and 3D contents. In
contrast to existing studies, this research attempts to evaluate user’s responses to the
emotional movie in 2D and 3D format.
The emotional movie is one of the emotionally intelligent contents (EICs). EICs refer
to the contents that recognize human emotions in real-time and provide visual and aural
feedback by offering appropriate stimuli so that users feel like they are communicating
with the contents [
]. In other words, the color, contrast, brightness or sound tempo of
the contents are changed dynamically based on users’ current emotional states to
provide more memorable or emotionally engaging experience to the users.
System design and implementation
The emotional movie system is built on top of the emotionally intelligent content
] to enable user preference specification of emotion. This framework allows
easy creation of emotional expressions so that the contents can be customized for users
depending on their own unique emotion rules. The emotion rules describe how the
contents are rendered visually and aurally in accordance with the individual’s emotional
The emotional movie system consists of the emotion recognition, the emotional movie
controller and the emotional movie player module. The emotion recognition detects the
users’ current emotional states in real-time by analyzing users’ autonomic nervous
system (ANS) physiological data. The emotional movie controller specifies the users’
emotion rules and selects the elements of the movie to change. It then responds to users’
current emotional states and controls the emotional movie player according to the
emotion rules. The emotional movie player is responsible for changing the visual and
aural effects of the movie according to the emotion rules.
Figure 1 shows the real-time user’s emotion recognition process by collecting,
normalizing and analyzing photoplethysmography (PPG), galvanic skin response (GSR) and
skin temperature (SKT) data. According to the range of the sensor values mapped to the
emotion rule base as shown in Table 1, users’ emotions are classified into nine categories
(i.e., excited, afraid, fatigued, calm, pleasant, aroused, unpleasant, relaxed, and neutral)
based on Russell’s two-dimensional emotion response model.
The PPG frequency and amplitude and the average values for the GSR and SKT
signals are used to extract features for emotion estimation, as studied in [
Pre-processing signal features are followed to compensate for noises and artefacts, consisting of
0.5–3.0 Hz band-pass filters for signals, and for absolute measures, a DC filter is used
including a 60 Hz notch filter for noise cancelling. A sliding window technique and time
dependent parameter (TDP) are also used for the real-time signal processing [
Normalization is adopted to minimize any individual difference in physiological data
readings by using the average value of the corresponding physiological data collected
from the neutral state to the current emotional state [
].The normalization method
is described by the following Eq. (1):
CPPG,GSR,SKT − NPPG,GSR,SKT
In Eq. (1), EPPG,GSR,SKT denotes the percentage increase or decrease of the PPG, GSR,
and SKT signals from the neural state (NPPG,GSR,SKT) to the current state (CPPG,GSR,SKT).
The threshold band is designed as a neutral range.
E[x] = lim 1 N
N →∞ N k=1
S[x] = lim 1
N →∞ N
(Xk (t) − E[x])2
TMax[Xn] = E[Xn] + S[Xn]
TMin[Xn] = E[Xn] − S[Xn]
In Eq. (2) and Eq. (3), E[x] denotes the mean values of the normalized PPG, SKT, and
GSR data and S[x] denotes the standard deviation of the normalized PPG, SKT, and GSR
data. In Eq. (4) and Eq. (5), the maximum and minimum range of the designed threshold,
TMax[Xn] and TMin[Xn], are calculated with a sliding window method.
The designed threshold band can classify the increase or decrease of physiological
signal patterns. The threshold range of normalized physiological signal patterns occurs at
the state of the following condition.
“+” symbol: Xk(t) > TMax[Xn]
“−” symbol: Xk(t) < TMin[Xn]
“0” symbol: TMin[Xn] ≤ Xk(t) ≤ TMax[Xn]
Finally, the normalized data are used to set the emotion rule base as shown in Table 1.
As a reference, the emotion rule base is defined in terms of nine emotional states. When
the physiological signal values are plotted as threshold range values, “+” and “−” denote
an “increase” or “decrease” in the normalized physiological signal values, respectively.
The “0” symbol stands for “no fluctuation”; in other words, the normalized physiological
signal values are affiliated with the threshold range.
Emotional movie controller
The emotional movie controller specifies the users’ emotion rules and the visual
elements (i.e., color, brightness and contrast) of the movie in order to effect users’ emotion.
It then responds to users’ current emotional states and controls the emotional movie
player using the emotion rules. Emotional contents allow users to specify personalized
emotion rule sets, which is stored in and retrieved from a database system [
Figure 2 shows the emotion rules describing how the emotional movies are rendered
visually depending on the users’ emotional state. The emotion rules were constructed
using a color theory [
]. In this theory, users feel smooth and lightweight when
there is high brightness and low contrast. They feel a heavy calm feeling when there is
low brightness and weak contrast.
In this research, the emotion rules cause the brightness to increase or decrease
gradually as the arousal state increases or decreases. The color saturation and contrast
increases or decreases as the valance state increases or decreases. The value of
saturation, brightness and contrast is adjusted by 3% level and used by the video rendering
filter for calculating the YUV values of the images. YUV is a color model in terms of one
luminance (Y) and two chrominance (UV).
Emotional states are used for the corresponding emotional movie controls mapped
to the preference of visual effects. For example, when the user’s emotional state is at
“Excited”, the brightness, contrast and saturation are increased. If the user’s emotion is
“Afraid”, the contrast and saturation are decreased while the brightness is increased.
Emotional movie player
The emotional movie player receives emotion control messages from the emotional
movie controller over a network and then makes filtering adjustments to the movie
settings as defined in the emotion rules. In this research, the color properties are used to
set the saturation (the intensity of a hue), brightness (the relative lightness) and contrast
(the difference between the darkest and lightest areas of a video) of the emotional movie.
For example, if users begin to be relaxed, the brightness is decreased, and if they feel
aroused, the brightness is increased. In addition, depending on the persistence of
pleasant and unpleasant emotion, color saturation and contrast effects were also decreased
and increased accordingly.
This experiment evaluates if there is a change in user emotion elicited by the
interactive natures of the emotional movie. In this experiment, the subjects watched the fearful
and joyful emotional movie clips shown in 2D and 3D format. The subject’s emotional
responses using physiological signals and their subjective feelings through a
posttest survey were gathered to evaluate if they felt more involved with the storyline and
showed more emotional responses.
A total of 12 college students majoring in computer science-related fields volunteered
to take part in this study. The average age of the subjects was 23.9 and of these 12, seven
were female and five were male. All subjects reported being familiar with the 3D, either
through playing 3D video games or watching 3D movies. However, most of them were
not familiar with the physiological sensors. Although none of them had ever seen the
actual movie, A Christmas Carol, they were familiar with the storyline and the main
character, Scrooge, through stories, books and other sources they had come across as
Figure 3 shows the emotional movie clips used in the experiment. The scenes were
chosen where a 3D effect would be most effective. Comparatively dark, frightening and
thrilling scenes were selected from the movie, A Christmas Carol, to create a 12 min
fearful clip (a–d). Similarly, comparatively light, warm and pleasant scenes were selected
to create a joyful clip of similar length (e–h).
a. Death of Scrooge’s partner, “Marley” (130 s).
b. Visit from Marley’s ghost (440 s).
c. Scrooge being chased by ghost hunters (110 s).
d. Scrooge slipping on a roof (40 s).
e. The first opening scene (130 s).
f. Journey with Christmas ghost (210 s).
g. Scrooge on Christmas morning (160 s).
h. Scrooge finally learning the joy of giving on Christmas (230 s).
Table 2 shows the average hue saturation value (HSV) color model values for each clip.
HSV is a color model that describes colors (hue) in terms of their shade (saturation or
amount of gray) and brightness (value). As shown in Table 2, the average brightness (V)
of the joyful scenes appears to be two times brighter than the fearful scenes.
Figure 4 shows the experimental setting. An active stereoscopic 46 inch Samsung TV
was used to show the 3D movie. The 2D clip was shown in full HD, while the 3D clip was
shown in a side-by-side stereoscopic image format and subjects wore shutter glasses to
receive the full 3D effect. The physiological signals were used to input emotion contents.
The PPG, GSR, SKT sensor signals were acquired at 500 Hz using the BioPac’s MP100
system during the test. The PPG signals were recorded with TSD203 sensors placed
on the index and ring fingers. The GSR signals were measured using two electrodes
which were attached to the index and middle fingers of the subject’s right hand with
Ag/AgCl gel materials. The SKT signals were measured with TSD202 sensors placed on
the thumb. Also, two channel electroencephalography (EEG) signals were measured at
In addition, one camera was used to record the subjects’ posture and movements and
one observer took notes on subjects’ behaviors during the test.
Figure 5 shows the experimental procedure. The total experimental procedure took
about 80 min. The subject was first given a brief explanation of the study and then they
consented to participate in the study. The pre-test survey asked for personal
information (i.e. age, gender, etc.) as well as questions regarding familiarity with 3D movies, and
familiarity with the movie, A Christmas Carol, and current emotional state. Then, the
physiological sensors were attached to the subject’s ear and fingers and initialized to the
The subjects’ physiological signals were measured for 2 min in a relaxed state to be
used as reference. The subjects were randomly assigned to view the 2D and 3D
emotional movie clips. Six of the 12 subjects were shown the 3D clips first and then 2D clips.
The remaining six subjects were shown the 2D clips first followed by the 3D clips. Finally,
the subjects were asked to fill out the post-test subjective evaluation questionnaires.
Based on two major categories of classical emotion theories: cognitive and somatic [
the subject’s emotional responses were measured by physiological signals and subjective
feeling. In the experiment, the subject’s physiological responses were measured by PPI
VLF/HF ratio, GSR startles, and EEG beta/alpha index. The subject’s affective responses
were measured by asking subjective feeling in post-test surveys.
The peak-to-peak interval (PPI) was detected from the PPG signals using peak
]. Contiguous PPI values were converted to a time series, and then linearly
interpolated to evaluate the true spectrum using an Fast Fourier transform (FFT) containing
the Hamming window. very-low frequency (VLF, in the range of 0.0033–0.04 Hz) and
high frequency (HF, in the range of 0.15–0.4 Hz) bands were then categorized. VLF/HF
ratio was used to identify the activity ratio of the sympathetic and parasympathetic
nervous system [
The GSR startles were used to measure the subject’s level of excitement. The GSR
startles were detected by smoothing the raw signal with a low pass filter, and a threshold is
applied to detect the steep slope associated with the rising edge of the startle response
by finding the local minimum preceding that point (onset) and the local maximum
following that point (peak).
The EEG beta/alpha indices were analyzed to measure the subject’s level of
concentration. The 2-channel brain waves were collected at the Fp1 and Fp2 position (i.e.,
prefrontal) according to the International 10–20 electrode placement system with a 512 Hz
sampling rate. A fast Fourier transform was employed to transform a raw signal to the
frequency domain. The alpha waves are the frequency range from 7.5 to 12.5 Hz and the
beta waves are the frequency range from 12.5 to 30 Hz.
In the post-test survey, subjects were asked to give their response on a scale from 1 to
5 for the questions regarding the overall feeling about the interactive emotional movies
and the effect of 3D emotional movie clips. On a more subjective level, the subjects were
asked to mark how they felt (out of one of the nine emotional states) as they watched
each clip. The subjects were also allowed to freely describe the scene that was most
memorable to them. Finally, the correlation rate was measured at which the emotional
responses of the subject’s physiological signals (objective measurement) were exactly
mapped onto the self-reported subjective feelings (subjective measurement).
Average Number of GSR Startles
D Avg. of E
H Avg. of
Fearful video clips
Fig. 7 Average of the number of GSR startles
Joyful video clips
In this experiment, the differences in 2D and 3D emotional movies composed of
fearful and joyful scenes are compared and analyzed. We evaluated the subject’s emotional
responses using their physiological responses as well as their self-reported affective
Figure 6 shows the average VLF/HF ratio of PPG signals while viewing the fearful and
joyful 2D and 3D emotional movie. An increase in high frequency components is
associated with activation of the parasympathetic nerve and emotional relaxation.
However, the paired t test results showed no significant mean differences between 2D and
3D for fearful and joyful video clips. The VLF/HF ratio was higher for the 2D fearful
clips (M = 0.99) than the 2D joyful clips (M = 0.59). It was higher for the 3D joyful clips
(M = 0.68) than the 3D fearful clips (M = 0.49).
Figure 7 shows the average GSR startles for each clip. The GSR startles indicates
how the given emotional movie induced user’s emotional responses. More number of
GSR startles means more arousals of excitement. Interestingly, greater number of GSR
startles was observed on 3D than 2D emotional movie, especially during the screening of
“E”, “G”, and “H” joyful clips (p < 0.05) and “C” and “D” fearful clips (p < 0.1).
The paired t test results showed a significant mean difference in GSR startles between
2D (M = 11.92) and 3D (M = 17.25) during the screening of joyful clips where the t value
was − 2.2921 with p = 0.021. There was also a significant mean difference in GSR
startles between 3D fearful (M = 9.25) and 3D joyful clips (M = 17.25) where the t value was
− 2.9008 with p = 0.00721.
Figure 8 show the average EEG beta/alpha index for each clip. The ratio of beta to
alpha index indicates the subject’s level of concentration. The average of EEG beta/alpha
index was slightly higher on 3D than 2D. However, the paired t-test showed no
significant mean difference between 2D and 3D for both fearful and joyful clips. Interestingly,
the statistically significant mean difference in EEG beta/alpha index between 2D and 3D
was found for parts of C, E and F clips.
The paired t-test revealed a significant mean difference in beta/alpha index between
2D (M = 2.429) and 3D (M = 3.552) with “being actively chased by ghost hunters” in clip
C (p < 0.05). Similarly, there was a significant difference between 2D and 3D with
“ballroom” and “the dog biting the fish and getting away” scenes in clip E (p < 0.05) and “the
appearance of the candle ghost”, “flying with the ghost to the past time”, and “talking with
friends in childhood” scenes in clip F (p < 0.05).
The post-test subjective evaluation results showed that ten subjects found the viewing
of the emotional movie to be more interesting than watching a regular movie. Eight
subjects thought the emotional movie to be more fun and two thought the emotional movie
was rather boring and tired. Three subjects (all of them experienced 3D first) reported
the visual fatigue, dizziness and discomfort of the 3D stereoscopic glasses. On the other
hand, five subjects (four of them experienced 2D first) reported the discomfort with the
emotional movie because they had difficulty to adapt to dynamic changes in color of the
The most frequently self-reported emotional state was “excited” or “aroused” after the
2D fearful clips and “excited” or “pleasant” after the 2D joyful clips. After the 3D fearful
clips, most subjects reported “excited” followed by a mixture of “aroused”, “unpleasant”
and “pleasant” state. On the other hand, most subjects reported “pleasant” state after the
3D joyful clips. As aforementioned, it is shown the increased occurrences of “pleasant”
state after the viewing of the joyful clips (than fearful clips) and 3D (than 2D).
Interestingly, the subjects recalled more scenes after the 3D emotional movie than 2D
emotional movie. Overall, the most memorable scenes were 3D ones with a very
perceptible depth, i.e. when Marley throws his chains, or very dynamic 3D ones, i.e. when
Scrooge flies to Christmas past with the candle ghost and when Scrooge is chased by
ghost hunters. These results matched with the cut-scenes where the EEG beta/alpha
index were the highest during the screening of the 3D emotional movie.
Correlation between subject’s physiological and subjective emotional responses
Figure 9 shows the average of the correlation between the subject’s emotional states
estimated by using physiological signals and the self-reported subjective feeling. The
correlation between the subject’s physiological responses and the subjective feeling was, on
average, about 45.83% match (2D fearful), 62.5% (2D joyful), 39.58% (3D fearful), and
68.75% (3D joyful). The paired t-test results showed a significant mean difference in the
correlation rate between 3D fearful video clips (M = 39.58%) and 3D joyful video clips
(M = 68.75%) where the t value was − 3.189 with p = 0.0043. We believed an accuracy of
62% to be high enough for the intuitive subject responses and the untrained
physiological responses since most subjects had never used physiological sensors before and none
of them were familiar with the emotional video system.
The emotionally intelligent contents tried to lead users to specific emotions by using
refined color masks, brightness, contrast, and other aural/visual effects. For example, a
first-person shooting game dynamically changed the color of the blood from a vivid red
to a more soft bright green in real-time when it detected a player being too focused on
the game [
]. Changing the color scheme of game artefacts is also found in a few
modern games to make them appear less violent for younger game players.
In prior work, it is known that 3D convey a higher sense of presence or realism but
also increases visual fatigue. A recent study found that 3D movies have little effect on
viewers’ emotional responses as compared to 2D movies [
], except for a thrilling scene
from The Polar Express. This particular scene had a great influence on user emotion,
with greater 3D effects than used in other films. While there are many studies
comparing 2D and 3D on user experience, such as visual fatigue, presence, and emotion [
little is known about the effect of emotional movie.
This research focused on studying user’s affective (subjective) and physiological
(objective) responses to the 3D emotional movie. Among various films, A Christmas
Carol (a 3D computer animation film released in 2009) was chosen because it delivers
the remarkable 3D effects that are similar to those in a theatrical 3D exhibition. Viewers
may feel like they can reach out and touch Marley’s heavy chains or fly over London or
travel through time to the past.
The movie clips are edited by fearful and joyful scenes in terms of both the storyline
and the color scheme, which also has an effect on user emotion. We expected that the
3D emotional movie provides a greater level of user experience than the 2D emotional
movie due to 3D effects. We also expected there to be more user affective and
physiological responses from the joyful clips than the fearful clips because a greater degree of
brightness creates a light-hearted feel.
The study results revealed that there was a little difference between 2D and 3D in
terms of physiological responses. A statistically significant mean difference was found
in number of GSR startles between 2D and 3D joyful clips. Also, a statistically
significant mean difference was found in the EEG beta/alpha index between 2D and 3D from
moment to moment, such as when Scrooge is being actively chased by ghost hunters,
when the candle ghost appears, and when Scrooge flies to Christmas past with the
Interestingly, the results revealed a difference between fearful and joyful clips during
the screening of the 3D emotional movie. The VLF/HF ratios of PPG and the beta/alpha
index of EEG were slightly increased with the 3D joyful clips as compared to the 3D
fearful clips. The number of GSR startles showed statistically significant increase with
the 3D joyful clips. The self-reported subjective feeling also showed more “pleasant”
with the joyful clips. This is somewhat similar to the findings from a study evaluating the
impact of color hue, brightness and saturation of websites on consumer emotions [
Moreover, the correlation between subject’s emotional states estimated by physiological
signals and self-reported subjective feeling was significantly higher in the 3D joyful clips
than the 3D fearful clips.
In recent years, there has been a rapid increase in the development of 3D contents and
technologies, such as 3D movies and virtual/augmented reality [
]. As interest in the
use of 3D technology increases, researchers have paid more attention to the effect of
3D contents on user experiences. There have been many research efforts showing that
the 3D contents can improve visual realism, presence, immersion and even performance
gains on some tasks, but at the same time, increase visual fatigue as compared to 2D
Over the past two decades, affective computing research has intensively studied
methods for emotion recognition [
], emotion expression [
] and emotion response
. However, only a few studies have looked at 3D emotional contents. Many existing
studies have strived to evaluate stimuli that evokes human affects, using the presentation
of emotional images and short film clips [
8–11, 34, 35
]. Typically, affective images or
short movie clips that are 1–3 min in length are selected to elicit target emotions in the
However, these approaches are rather passive since they only measure user emotions
elicited by the contents and do not reflect users’ affective states into the contents. In this
research, we considered the emotionally intelligent contents that respond interactively
to users’ current affective states and the visual effects of the contents that are
dynamically changed depending on the users’ emotional changes.
We have conducted a user study on the emotional movies where the saturation,
contrast and brightness of the movie were increased or decreased by user’s emotional
states. The objective was to study users’ affective (subjective) and physiological
(objective) responses to the interactive natures of the emotional movies to find out if users feel
more involved with the storyline and/or if they evoke more emotional responses. The
edited footage of Disney’s A Christmas Carol, going from fearful and joyful scenes was
presented in both 2D and 3D formats.
Overall, there was no statistically significant mean difference between 2D and 3D
emotional movies. However, a statistically significant difference was found in the GSR
startles between the 2D and 3D joyful clips as well as in the EEG beta/alpha index between
some short 2D and 3D scenes. As aforementioned, users showed more responses to 3D
bright joyful clips than to 3D dark fearful clips.
User emotion is a complex phenomenon. The factors that affect user emotions are
varied. This research is the first attempt to analyze what would affect users’ emotional states
in the emotional movie, i.e., the effect of 2D or 3D, and the role of fearful or joyful clips
(such as colors/tones of light in the scenes). However, we need to further compare and
study regular 3D contents with 3D emotional contents to determine whether the users’
emotional states are elicited by real-time changes of visual effects in the emotional
contents or by the colors or tones of the scenes.
First author is DK, contributing to the Emotion Recognition section. Second author is YC, contributing to the Emotionally
Movie system section. Corresponding author is KSP, contributing to all sections of the manuscript. All authors read and
approved the final manuscript.
1 Department of Intelligent Engineering Informatics for Human, College of Future Convergence Engineering,
Sangmyung University, 20 Honjimun 2-gil, Jongno-gu, Seoul 03016, South Korea. 2 Department of Computer Science, College
of Future Convergence Engineering, Sangmyung University, 20 Hongjimun 2-gil, Jongno-gu, Seoul 03016, South Korea.
3 Department of Applied Computer Engineering, College of Software Convergence, Dankook University, 152 Jukjeon-ro,
Suji-gu, Yongin-si, Gyeonggi-do 16890, South Korea.
The authors declare that they have no competing interests.
Availability of data and materials
Ethics approval and consent to participate
IRB form is approved by Sangmyung University. All participants are from Sangmyung University. Written informed
consent was obtained from the subjects for the publication of this article and any accompanying images.
Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.
1. McIntire JP , Havig PR , Geiselman EE ( 2014 ) Stereoscopic 3D displays and human performance: a comprehensive review . Displays 35 ( 1 ): 18 - 26 . https://doi.org/10.1016/j.displa. 2013 . 10 .004
2. Li J , Barkowsky M , Callet P-L ( 2014 ) Visual discomfort of stereoscopic 3D videos: influence of 3D motion . Displays 35 ( 1 ): 49 - 57 . https://doi.org/10.1016/j.displa. 2014 . 01 .002
3. Kim DW , Yoo JS , Seo YH ( 2013 ) Qualitative analysis of individual and composite content factors of stereoscopic 3D video causing visual discomfort . Displays 34 ( 3 ): 223 - 240 . https://doi.org/10.1016/j.displa. 2013 . 07 .002
4. Chen C , Li K , Wu Q , Wang H , Qian Z , Sudlow G ( 2013 ) EEG-based detection and evaluation of fatigue cause by watching 3DTV . Displays 34 ( 2 ): 81 - 88 . https://doi.org/10.1016/j.displa. 2013 . 01 .002
5. Yano S , Ide S , Mitsuhashi T , Thwaites H ( 2002 ) A study of visual fatigue and visual comfort for 3D HDTV/HDTV images . Displays 23 ( 4 ): 191 - 201 . https://doi.org/10.1016/S0141- 9382 ( 02 ) 00038 - 0
6. Kim CJ , Park S , Won M , Whang M , Lee EC ( 2013 ) Autonomic nervous system responses can reveal visual fatigue induced by 3D displays . Sensors 13 ( 10 ): 13054 - 13062 . https://doi.org/10.3390/s131013054
7. Sakamoto K , Asahara S , Sakashita S , Yamashita K , Okada A ( 2012 ) Influence of 3DTV video contents on physiological measurements of emotional state . IEEE Int Symp Consum Electron . https://doi.org/10.1109/ISCE. 2012 .6241712
8. Sakamoto K , Asahara S , Yamashita K , Okada A ( 2011 ) Physiological and psychological measurement of emotional state using various types of video content during TV viewing . IEEE Int Symp Consum Electron . https://doi. org/10.1109/ISCE. 2011 .5973780
9. Money A-G , Aguis H ( 2009 ) Analysing user physiological responses for affective video summarization . Displays 30 ( 2 ): 59 - 70 . https://doi.org/10.1016/j.displa. 2008 . 12 .003
10. Hanjalic A , Xu LQ ( 2005 ) Affective video content representation and modeling . IEEE Trans Multimedia 7 ( 1 ): 143 - 154 . https://doi.org/10.1109/TMM. 2004 .840618
11. Bride DL , Crowell SE , Baucom BR , Kaufman EA , O'Connor CG , Skidmore CR , Yaptangco M ( 2014 ) Testing the effectiveness of 3D film for laboratory-based studies of emotion . PLoS ONE 9 :e105554. https://doi.org/10.1371/journ al. pone.0105554
12. Park KS , Cho Y , Kim DK ( 2014 ) A framework for the creating, expressing and sharing of user's emotion . Int J Multimedia Ubiquitous Eng . 9 ( 1 ): 425 - 442 . https://doi.org/10.14257/ijmue. 2014 . 9 .1. 39
13. Verschuere B , Crombez G , Koster EHW ( 2001 ) The international affective picture system: a cross cultural validation study . Psychologica Belgica 41 ( 4 ): 205 - 217
14. Park JS , Kim JH , Oh YH ( 2009 ) Feature vector classification based speech emotion recognition for service robots . IEEE Trans Consum Electron 55 : 1590 - 1596 . https://doi.org/10.1109/TCE. 2009 .5278031
15. Pelet JE , Conway CM , Papadopoulou P , Limayem M ( 2015 ) Chromatic scales on our eyes: how user trust in a website can be altered by color via emotion . Adv Intell Syst Comput 205 : 111 - 121 . https://doi.org/10.1007/978-3- 642 -37317 -6_ 10
16. Cho EJ , Kweon SH , Cho BC ( 2010 ) A study of presence perception tendency by genres on 3DTV . Korean J Broadcast Telecommun Stud 24 : 253 - 292
17. Baranowski AM , Keller K , Neumann J , Hecht H ( 2016 ) Genre-dependent effects of 3D film on presence, motion sickness, and protagonist perception . Displays 44 : 53 - 59 . https://doi.org/10.1016/j.displ. 2016 . 06 .004
18. Rooney B , Hennessy E ( 2013 ) Actually in the cinema: a field study comparing real 3D and 2D movie patrons' attention, emotion, and film satisfaction . Media Psychol 16 : 441 - 460 . https://doi.org/10.1080/15213269. 2013 .838905
19. Yang S , Sheedy JE ( 2011 ) Effects of vergence and accommodative responses on viewer's comfort in viewing 3D stimuli . In: Woods AJ , Holliman NS , Dodgson NA (eds) Proceedings of SPIE 7863, stereoscopic displays and applications XXII , San Francisco, CA, USA, 23 January. SPIE, Bellingham. http://dx.doi.org/10.1117/12.872546
20. Rajae-Joordens RJE ( 2008 ) Measuring experiences in gaming and TV applications: investigating the added value of a multi-view auto-stereoscopic 3D display . In: Westerink JHDM , Ouwerkerk M , Overbeek TJM , Pasveer WF , de Ruyter B (eds) Probing experience: from assessment of user emotions and behaviour to development of products , vol 8 . Springer, Dordrecht, pp 77 - 90 . https://doi.org/10.1007/978-1- 4020 -6593- 4 _ 7
21. Kim M , Park KS , Kim D , Cho Y ( 2011 ) Emotional intelligent contents: expressing user's own emotion within contents . Lect Notes Comput Sci 6972 : 391 - 394 . https://doi.org/10.1007/978-3- 642 -24500-8_ 49
22. Kim DK , Kim JH , Lee EC , Whang MC , Cho Y ( 2011 ) Interactive emotional content communications system using portable wireless biofeedback device . IEEE Trans Consum Electron 57 ( 4 ): 1929 - 1936 . https://doi.org/10.1109/ TCE. 2011 .6131173
23. Kim JH , Whang MC , Kim YJ , Woo JC ( 2008 ) The study on emotion recognition by time-dependent parameters of autonomic nervous response . Sci Emot Sensib 11 ( 4 ): 637 - 643
24. Kim DK , Ahn SM , Park SM , Whang MC ( 2013 ) Interactive emotional lighting system using physiological signals . IEEE Trans Consum Electron 59 ( 4 ): 765 - 771 . https://doi.org/10.1109/TCE. 2013 .6689687
25. Liechti O , Ichikawa T ( 1999 ) A digital photography framework supporting social interaction and affective awareness . Lect Notes Comput Sci 1707 : 186 - 192 . https://doi.org/10.1007/3-540-48157-5_ 18
26. Kim JH ( 2011 ) The key to make everything look better, color . Gilbut, Seoul, pp 97 - 105
27. Lopatovska I , Arapakis I ( 2011 ) Theories, methods and current research on emotions in library and information science, information retrieval and human-computer interaction . Inf Process Manag 47 ( 4 ): 575 - 592 . https://doi. org/10.1016/j.ipm. 2010 . 09 .001
28. Kim J , Shin H ( 2016 ) Simple and robust realtime QRS Detection algorithm based on spatiotemporal characteristic of the QRS complex . PLoS ONE 11 :e0150144. https://doi.org/10.1371/journal.pone.0150144
29. Pomeranz B , Macaulay RJB , Caudill MA , Kutz I , Adam D , Gordon D , Kilborn KM , Benson H ( 1985 ) Assessment of autonomic function in humans by heart rate spectral analysis . Am J Physiol 248 ( 1 Pt 2 ): H151 - H153
30. Chen CY , Ke MD , Wu PJ , Kuo CD , Pong BJ , Lai YY ( 2014 ) The influence of polarized 3D display on autonomic nervous activities . Displays 35 ( 4 ): 196 - 201 . https://doi.org/10.1016/j.displa. 2014 . 05 .001
31. Kim M , Park KS , Kim D , Cho Y ( 2012 ) Open source library based emotional game engine and emotion sharing game contents . J Korean Soc Comput Game . 25 : 15 - 24
32. Jeon J , Hong M , Yi M , Chun J , Kim JS , Choi YJ ( 2016 ) Interactive authoring tool for mobile augmented reality content . J Inf Process Syst 12 ( 4 ): 612 - 630 . https://doi.org/10.3745/JIPS.02.0048
33. Baniya BK , Lee J ( 2017 ) Rough set-based approach for automatic emotion classification of music . J Inf Process Syst 13 ( 2 ): 400 - 416 . https://doi.org/10.3745/JIPS.04.0032
34. Frantzidis CA , Bratsas C , Klados MA , Konstantinidis E , Lithari CD , Vivas AB , Bamidis PD ( 2010 ) On the classification of emotional biosignals evoked while viewing affective pictures: an integrated data-mining-based approach for healthcare applications . IEEE Trans Inf Technol Biomed 14 ( 2 ): 309 - 318 . https://doi.org/10.1109/TITB. 2009 .2038481
35. Kory J , D'Mello SK ( 2005 ) Affective elicitation for affective computing . In: Calvo R, D'Mello S , Gratch J , Kappas A (eds) The oxford handbook of affective computing . Oxford University Press, Oxford, pp 371 - 383 . https://doi.org/10.1093/ oxfordhb/9780199942237.013.001