Intermodal event files: integrating features across vision, audition, taction, and action
Sharon Zmigrod
0
Michiel Spap
0
Bernhard Hommel
0
0
S. Zmigrod (&) M. Spap B. Hommel Department of Psychology, Cognitive Psychology Unit, Leiden University Institute for Psychological Research and Leiden Institute for Brain and Cognition
, Postbus 9555, 2300 RB Leiden,
The Netherlands
Understanding how the human brain integrates features of perceived events calls for the examination of binding processes within and across diVerent modalities and domains. Recent studies of feature-repetition eVects have demonstrated interactions between shape, color, and location in the visual modality and between pitch, loudness, and location in the auditory modality: repeating one feature is beneWcial if other features are also repeated, but detrimental if not. These partial-repetition costs suggest that cooccurring features are spontaneously bound into temporary event Wles. Here, we investigated whether these observations can be extended to features from diVerent sensory modalities, combining visual and auditory features in Experiment 1 and auditory and tactile features in Experiment 2. The same types of interactions, as for unimodal feature combinations, were obtained including interactions between stimulus and response features. However, the size of the interactions varied with the particular combination of features, suggesting that the salience of features and the temporal overlap between feature-code activations plays a mediating role.
-
Human perception is multisensory, that is, we get to know
our environment through multiple sensory modalities. The
existence of multisensory perception raises the question of
how the diVerent sensory modalities features we process
are integrated into coherent, uniWed representations. For
example, eating an apple requires making sense of visual
features such as the shape, color, and location of the fruit; a
distinctive bite sound pattern of a particular pitch and
loudness; a particular texture, weight, and temperature of the
apple; and chemical features characterizing the apples
taste and smell. These features are processed in distinct
cortical regions and along diVerent neural pathways (e.g.,
Goldstein, 2007), so that some mechanism is needed to
bind them into a coherent perceptual representationso as
to solve what is known as the binding problem
(Treisman, 1996). In the last decade, the investigation of binding
processes has focused on visual perception (e.g., Allport,
Tipper, & Chmiel 1985; Treisman & Gelade, 1980) and
only recently been extended to the auditory domain (e.g.,
Hall, Pastore, Acker, & Huang 2000; Takegata, Brattico,
Tervaniemi, Varyagina, Ntnen, & Winkler 2005).
However, real objects are rarely deWned and perceived in just
one isolated modality, but rather call for interactions among
many sensory modalities. Therefore, an eYcient feature
binding mechanism should operate in a multi-modal
manner and bind features regardless of their modality.
In recent years, diVerent research strategies were
introduced to study multisensory perception. Some studies
created situations of perceptual conXict such that two sensory
modalities received incongruent information, which often
produced perceptual illusions and, occasionally, even
longer lasting after eVects. A classic example is the
McGurk eVect in which vision changes speech perception:
an auditory /ba/ sound is perceived as /da/ if paired with a
visual lip movement saying /ga/ (McGurk & MacDonald,
1976). An additional audio-visual example is the
ventriloquism eVect: people mislocate sound sources after being
exposed to concurrent auditory and visual stimuli appearing
at disparate locations (e.g., Bertelson, Vroomen, de Gelder,
& Driver 2000; Vroomen, Bertelson, & de Gelder 2001).
Another, more recently discovered illusion is the
auditoryvisual double Xash eVect in which a single visual Xash is
perceived as multiple Xashes when accompanied by
sequences of auditory beeps (Shams, Kamitani, & Shimojo
2000). This illusion was also found in the auditory-tactile
domain, where a single tactile stimulus leads to the
perception of multiple tactile events if accompanied by tone
sequences (Htting & Rder, 2004). These and other
studies in the multisensory domain provide evidence for on-line
interactions between diVerent sensory modalities, but they
have not led to a comprehensive understanding of how the
brain integrates those diVerent features into coherent
perceptual structures.
The purpose of the present study was to investigate
multi-modal feature integration through the analysis of
feature-repetition eVects or, more precisely, of interactions
between them. As Kahneman, Treisman, and Gibbs (1992),
and many others since then, have shown, repeating a visual
stimulus facilitates performance but more so if its location
is also repeated. Further studies have demonstrated
interactions between repetition eVects for various visual and
auditory features. For instance, repeating a visual shape
improves performance if its color is also repe (...truncated)