The audiovisual structure of onomatopoeias: An intrusion of real-world physics in lexical creation

PLOS ONE, Nov 2019

Sound-symbolic word classes are found in different cultures and languages worldwide. These words are continuously produced to code complex information about events. Here we explore the capacity of creative language to transport complex multisensory information in a controlled experiment, where our participants improvised onomatopoeias from noisy moving objects in audio, visual and audiovisual formats. We found that consonants communicate movement types (slide, hit or ring) mainly through the manner of articulation in the vocal tract. Vowels communicate shapes in visual stimuli (spiky or rounded) and sound frequencies in auditory stimuli through the configuration of the lips and tongue. A machine learning model was trained to classify movement types and used to validate generalizations of our results across formats. We implemented the classifier with a list of cross-linguistic onomatopoeias simple actions were correctly classified, while different aspects were selected to build onomatopoeias of complex actions. These results show how the different aspects of complex sensory information are coded and how they interact in the creation of novel onomatopoeias.

A PDF file should load here. If you do not see its contents the file may be temporarily unavailable at the journal website or you do not have a PDF plug-in installed and enabled in your browser.

Alternatively, you can download the file locally and open with any standalone PDF reader:

https://journals.plos.org/plosone/article/file?id=10.1371/journal.pone.0193466&type=printable

The audiovisual structure of onomatopoeias: An intrusion of real-world physics in lexical creation

March The audiovisual structure of onomatopoeias: An intrusion of real-world physics in lexical creation Alan Taitz 0 1 M. Florencia Assaneo 0 1 Natalia Elisei 1 Mo nica TrÂõpodi 1 Laurent Cohen 1 Jacobo D. Sitt 1 Marcos A. Trevisan 0 1 0 Department of Physics, IFIBA-University of Buenos Aires , Buenos Aires , Argentina , 2 Department of Psychology, New York University, New York, United States of America, 3 Medicine School, University of Buenos Aires , Buenos Aires , Argentina , 4 Department of Linguistics, University of Buenos Aires , Buenos Aires , Argentina , 5 INSERM U1127, Institut du Cerveau et de la Moelle E pinière, Paris, France, 6 CNRS UMR 7225, Institut du Cerveau et de la Moelle E pinière , Paris, France, 7 Sorbonne Universite s, UPMC Univ Paris 06, Paris, France, 8 AP-HP , Groupe Hospitalier Pitie -Salpêtrière, Departament of Neurology , Paris , France 1 Editor: Niels O. Schiller, Leiden University , NETHERLANDS Sound-symbolic word classes are found in different cultures and languages worldwide. These words are continuously produced to code complex information about events. Here we explore the capacity of creative language to transport complex multisensory information in a controlled experiment, where our participants improvised onomatopoeias from noisy moving objects in audio, visual and audiovisual formats. We found that consonants communicate movement types (slide, hit or ring) mainly through the manner of articulation in the vocal tract. Vowels communicate shapes in visual stimuli (spiky or rounded) and sound frequencies in auditory stimuli through the configuration of the lips and tongue. A machine learning model was trained to classify movement types and used to validate generalizations of our results across formats. We implemented the classifier with a list of cross-linguistic onomatopoeias simple actions were correctly classified, while different aspects were selected to build onomatopoeias of complex actions. These results show how the different aspects of complex sensory information are coded and how they interact in the creation of novel onomatopoeias. - Data Availability Statement: All relevant data are within the paper and its Supporting Information files. Funding: The research reported in this work was partially funded by the Concejo Nacional de Investigaciones CientÂõficas y TeÂcnicas (CONICET) to AT and MAT; The University of Buenos Aires (UBA); ECOS A11S01 scientific collaboration program (France-Argentina) to MFA, LC, JDS, and MAT; NIH through R01-DC-012859 to MAT, and the program ªInvestissements d'avenirº (ANR-10 Introduction The arbitrary nature of the linguistic sign has been an idealized notion of modern linguistics that served to explore the unlimited expressive power of language [ 1 ]. As research evolved, the fully arbitrary nature of the link between form and meaning has been called into question, opening the scientific exploration of this relationship. The results of those investigations show that nonarbitrary associations are not limited to exceptional cases or to specific word classes: a IAIHU-06) to JS. The funders had no role in study design, data collection and analysis, decision to publish, or preparation of the manuscript. striking demonstration of this comes from a statistical analysis performed over two-thirds of the world's languages, revealing that unrelated languages use the same sounds for specific referents [ 2 ]. Iconicity is a prominent form of non-arbitrariness, in which different aspects of the form and the meaning of words are related by perceptuomotor analogies [ 3 ]. Onomatopoeias are privileged objects to study iconic properties of spoken words. Just like any other word, onomatopoeias are embedded in the language and have to adapt to the local phonology, assuming arbitrary properties. However, they also they tend to maximize the similarity between speech sounds and the sounds of the actions they represent, preserving parts of the onomatopoeic structure across languages [ 4 ]. Beyond sound imitation stand the mimetic words, a more general class used to express actions where sound is not essential [ 5 ]. It has been suggested that these words, found in Japanese, are related to the interaction between the body and the linguistic sound system: mimetic words use sounds to imitate sensations including body movements, touch, vision, smell, taste, and sound [ 6 ]. Interestingly, this kind of words were found in many other languages [7±9]. The concept of ideophone was then coined to characterize 'a word, often onomatopoeic, which describes a predicate, qualificative or adverb in respect to manner, color, sound, smell, action, state or intensity' [ 10 ]. For instance, operations like lengthening and reduplication tend to evoke repetition and multiplicity, while monosyllabic forms tend to evoke unitary events [ 11 ]. Ideophones rise from the rest of the words to depict sensory imagery; beyond the mere imitation of soun (...truncated)


This is a preview of a remote PDF: https://journals.plos.org/plosone/article/file?id=10.1371/journal.pone.0193466&type=printable

Alan Taitz, M. Florencia Assaneo, Natalia Elisei, Mónica Trípodi, Laurent Cohen, Jacobo D. Sitt, Marcos A. Trevisan. The audiovisual structure of onomatopoeias: An intrusion of real-world physics in lexical creation, PLOS ONE, 2018, Volume 13, Issue 3, DOI: 10.1371/journal.pone.0193466