Intermodal event files: integrating features across vision, audition, taction, and action

Psychological Research, Sep 2009

Understanding how the human brain integrates features of perceived events calls for the examination of binding processes within and across different modalities and domains. Recent studies of feature-repetition effects have demonstrated interactions between shape, color, and location in the visual modality and between pitch, loudness, and location in the auditory modality: repeating one feature is beneficial if other features are also repeated, but detrimental if not. These partial-repetition costs suggest that co-occurring features are spontaneously bound into temporary event files. Here, we investigated whether these observations can be extended to features from different sensory modalities, combining visual and auditory features in Experiment 1 and auditory and tactile features in Experiment 2. The same types of interactions, as for unimodal feature combinations, were obtained including interactions between stimulus and response features. However, the size of the interactions varied with the particular combination of features, suggesting that the salience of features and the temporal overlap between feature-code activations plays a mediating role.

A PDF file should load here. If you do not see its contents the file may be temporarily unavailable at the journal website or you do not have a PDF plug-in installed and enabled in your browser.

Alternatively, you can download the file locally and open with any standalone PDF reader:

https://link.springer.com/content/pdf/10.1007%2Fs00426-008-0163-5.pdf

Intermodal event files: integrating features across vision, audition, taction, and action

Sharon Zmigrod 0 Michiel Spap 0 Bernhard Hommel 0 0 S. Zmigrod (&) M. Spap B. Hommel Department of Psychology, Cognitive Psychology Unit, Leiden University Institute for Psychological Research and Leiden Institute for Brain and Cognition , Postbus 9555, 2300 RB Leiden, The Netherlands Understanding how the human brain integrates features of perceived events calls for the examination of binding processes within and across diVerent modalities and domains. Recent studies of feature-repetition eVects have demonstrated interactions between shape, color, and location in the visual modality and between pitch, loudness, and location in the auditory modality: repeating one feature is beneWcial if other features are also repeated, but detrimental if not. These partial-repetition costs suggest that cooccurring features are spontaneously bound into temporary event Wles. Here, we investigated whether these observations can be extended to features from diVerent sensory modalities, combining visual and auditory features in Experiment 1 and auditory and tactile features in Experiment 2. The same types of interactions, as for unimodal feature combinations, were obtained including interactions between stimulus and response features. However, the size of the interactions varied with the particular combination of features, suggesting that the salience of features and the temporal overlap between feature-code activations plays a mediating role. - Human perception is multisensory, that is, we get to know our environment through multiple sensory modalities. The existence of multisensory perception raises the question of how the diVerent sensory modalities features we process are integrated into coherent, uniWed representations. For example, eating an apple requires making sense of visual features such as the shape, color, and location of the fruit; a distinctive bite sound pattern of a particular pitch and loudness; a particular texture, weight, and temperature of the apple; and chemical features characterizing the apples taste and smell. These features are processed in distinct cortical regions and along diVerent neural pathways (e.g., Goldstein, 2007), so that some mechanism is needed to bind them into a coherent perceptual representationso as to solve what is known as the binding problem (Treisman, 1996). In the last decade, the investigation of binding processes has focused on visual perception (e.g., Allport, Tipper, & Chmiel 1985; Treisman & Gelade, 1980) and only recently been extended to the auditory domain (e.g., Hall, Pastore, Acker, & Huang 2000; Takegata, Brattico, Tervaniemi, Varyagina, Ntnen, & Winkler 2005). However, real objects are rarely deWned and perceived in just one isolated modality, but rather call for interactions among many sensory modalities. Therefore, an eYcient feature binding mechanism should operate in a multi-modal manner and bind features regardless of their modality. In recent years, diVerent research strategies were introduced to study multisensory perception. Some studies created situations of perceptual conXict such that two sensory modalities received incongruent information, which often produced perceptual illusions and, occasionally, even longer lasting after eVects. A classic example is the McGurk eVect in which vision changes speech perception: an auditory /ba/ sound is perceived as /da/ if paired with a visual lip movement saying /ga/ (McGurk & MacDonald, 1976). An additional audio-visual example is the ventriloquism eVect: people mislocate sound sources after being exposed to concurrent auditory and visual stimuli appearing at disparate locations (e.g., Bertelson, Vroomen, de Gelder, & Driver 2000; Vroomen, Bertelson, & de Gelder 2001). Another, more recently discovered illusion is the auditoryvisual double Xash eVect in which a single visual Xash is perceived as multiple Xashes when accompanied by sequences of auditory beeps (Shams, Kamitani, & Shimojo 2000). This illusion was also found in the auditory-tactile domain, where a single tactile stimulus leads to the perception of multiple tactile events if accompanied by tone sequences (Htting & Rder, 2004). These and other studies in the multisensory domain provide evidence for on-line interactions between diVerent sensory modalities, but they have not led to a comprehensive understanding of how the brain integrates those diVerent features into coherent perceptual structures. The purpose of the present study was to investigate multi-modal feature integration through the analysis of feature-repetition eVects or, more precisely, of interactions between them. As Kahneman, Treisman, and Gibbs (1992), and many others since then, have shown, repeating a visual stimulus facilitates performance but more so if its location is also repeated. Further studies have demonstrated interactions between repetition eVects for various visual and auditory features. For instance, repeating a visual shape improves performance if its color is also repe (...truncated)


This is a preview of a remote PDF: https://link.springer.com/content/pdf/10.1007%2Fs00426-008-0163-5.pdf

Sharon Zmigrod, Michiel Spapé, Bernhard Hommel. Intermodal event files: integrating features across vision, audition, taction, and action, Psychological Research, 2009, pp. 674-684, Volume 73, Issue 5, DOI: 10.1007/s00426-008-0163-5