Iconic gestures prime related concepts: An ERP study

YING CHOON WU 0 SEANA COULSON 0 0 University of California , San Diego, La Jolla, California To assess priming by iconic gestures, we recorded EEG (at 29 scalp sites) in two experiments while adults watched short, soundless videos of spontaneously produced, cospeech iconic gestures followed by related or unrelated probe words. In Experiment 1, participants classified the relatedness between gestures and words. In Experiment 2, they attended to stimuli, and performed an incidental recognition memory test on words presented during the EEG recording session. Event-related potentials (ERPs) time-locked to the onset of probe words were measured, along with response latencies and word recognition rates. Although word relatedness did not affect reaction times or recognition rates, contextually related probe words elicited less-negative ERPs than did unrelated ones between 300 and 500 msec after stimulus onset (N400) in both experiments. These findings demonstrate sensitivity to semantic relations between iconic gestures and words in brain activity engendered during word comprehension. - During discourse, speakers use hand and body movements to depict conceptual content salient to their talk. Individuals have traced an oval shape in the air while describing a platter, for example, or demonstrated running legs by wiggling two fingers (McNeill, 1992; Wu & Coulson, 2005). It has been proposed that movements such as these, known as iconic gestures (McNeill, 1992), affect listener comprehension. A number of behavioral studies have demonstrated listener sensitivity to information conveyed in gestures (Alibali, Flevares, & Goldin-Meadow, 1997; Cassell, McNeill, & McCullough, 1999; GoldinMeadow & Sandhofer, 1999), as well as improved comprehension of spoken discourse when speakers gestures are visible (Beattie & Shovelton, 1999, 2002; Rogers, 1978; Valenzeno, Alibali, & Klatzky, 2003). Measuring event-related potentials (ERPs), researchers have also shown differences in brain activity elicited by words presented with congruent as opposed to incongruent gestures, or with no gestures (Kelly, Kravitz, & Hopkins, 2004). These findings suggest that iconic gestures are analyzed for meaning, and can produce measurable effects on observer comprehension. Recent research has investigated commonalities in semantic processes mediating the comprehension of iconic gestures and the comprehension of more conventional visual representations such as pictures. Picture probes, for example, have been shown to elicit more-negative ERPs around 300 msec (N300) and 400 msec (N400) poststimulus when they are preceded by unrelated picture primes in comparison with related ones (Barrett & Rugg, 1990; Ganis, Kutas, & Sereno, 1996). The N400 relatedness effect elicited by pictures is similar to the classic N400 elicited by words. Originally discovered in response to sentence-final words, the lexical N400 was described as a negative-going deflection of the ERP waveform peaking between 300 and 500 msec poststimulus, with an enhanced amplitude for incongruous items in comparison with congruous ones (Kutas & Hillyard, 1980, 1984). Subsequently, similar effects were obtained in congruency manipulations involving a number of different stimulus types, including written and spoken word pairs (Holcomb & Neville, 1990), photographs (McPherson & Holcomb, 1999), and videotaped actions (Sitnikova, Kuperberg, & Holcomb, 2003). Further, American Sign Language (ASL) hand signs (Neville et al., 1997), which recruit more bilateral cortical resources than spoken language (Bavelier et al., 1998), and emblematic gestures (e.g., thumbs up; Gunter & Bach, 2004) have also elicited N400-like activity. These findings suggest that the N400 class of negativitiesalthough it probably results from overlapping, but nonidentical, neural generatorsis a brain response triggered by meaningful stimuli. Because the amplitude of the N400 is inversely correlated with the degree to which an item is expected in its context (Kutas & Hillyard, 1984; Van Petten & Kutas, 1990), this brainwave component is generally thought to index the integration of incoming semantic information into a higher order mental model. To test for N400-like responses to gestures, we recorded ERPs while participants watched spontaneously produced iconic gestures preceded by either congruous or incongruous cartoon contexts (Wu, 2005; Wu & Coulson, 2005). In comparison with congruous trials, incongruous gestures elicited more-negative ERPs between 450 and 600 msec poststimulus (gesture N450). This effect displayed a time course and polarity similar to those of the N400 family of negativities, as well as similar eliciting conditions. These data suggest that, like words and pictures, iconic gestures also engage meaning-based representations that are integrated with other contextually active information. Further support for this view was uncovered by extracting static freeze-frames from gesture videos, and pairing them with congruent or incongruent cartoon contexts (Wu, 2005). In addition to exhibiting enhanced N450, incongruent freeze-frames also elicited more-negative ERPs between 300 and 400 msec (N300). Like the N400, the N300 is sensitive to manipulations of relatedness between images and prior context. However, because the N300 has only been observed in response to pictures and photos, it is thought to reflect the activation of image-specific conceptual representations. The finding that incongruent static gestures elicited a larger N300 suggests that understanding these items semiotic features (e.g., hand shape, location, and orientation) recruits similar comprehension processes as well. It is possible that dynamic gestures did not yield N300 effects because processes indexed by the N300 might become activated slightly later in response to moving, visually complex stimuli than they would in response to static objects, overlapping with processes indexed by the N450. The present study investigates whether iconic gestures activate meaning-based representations in the absence of supporting linguistic context. Because iconic gestures are not part of a conventionalized symbolic system, it has been argued that their meaning is determined largely by speech accompanying them (Krauss, Morell-Samuels, & Colasante, 1991). However, given evidence that understanding gestures engages semantic processes analogous to those recruited by pictures, people may be capable of integrating the semiotic features of gestures with stored knowledge about their referents, even in the absence of contextual support. To test this hypothesis, we recorded ERPs while healthy adults watched spontaneously produced iconic gestures followed by probe words. In Experiment 1, participants classified probes as related or unrelated; in Experiment 2, they attended to stimuli and completed a test of incidental probe recognition afterward. If gestures activate stored knowledge about the phenomena they depict, evidence o (...truncated)