When hearing the bark helps to identify the dog: Semantically-congruent sounds modulate the identification of masked pictures

被引:149
作者
Chen, Yi-Chuan [1 ]
Spence, Charles [1 ]
机构
[1] Univ Oxford, Dept Expt Psychol, Crossmodal Res Lab, Oxford OX1 3UD, England
关键词
Semantic congruency; Object identification; Audiovisual; Multisensory; SELECTIVE ATTENTION; REACTION-TIME; AUDIOVISUAL INTEGRATION; UNCONSCIOUS PERCEPTION; OBJECT RECOGNITION; VISUAL INFORMATION; CONCEPTUAL MASKING; UNITY ASSUMPTION; LOW-LEVEL; WORD;
D O I
10.1016/j.cognition.2009.10.012
中图分类号
B84 [心理学];
学科分类号
04 ; 0402 ;
摘要
We report a series of experiments designed to assess the effect of audiovisual semantic congruency on the identification of visually-presented pictures. Participants made unspeeded identification responses concerning a series of briefly-presented, and then rapidly-masked. Pictures. A naturalistic sound was sometimes presented together with the picture at a stimulus onset asynchrony (SOA) that varied between 0 and 533 ms (auditory lagging). The sound could be semantically congruent, semantically incongruent, or else neutral (white noise) with respect to the target picture. The results showed that when the onset of the picture and sound occurred simultaneously, a semantically-congruent sound improved, whereas a semantically-incongruent sound impaired, participants' picture identification performance, as compared to performance in the white-noise control condition. A significant facilitatory effect was also observed at SOAs of around 300 ms, whereas no such semantic congruency effects were observed at the longest interval (533 ms). These results therefore suggest that the neural representations associated with visual and auditory stimuli can interact in a shared semantic system. Furthermore, this crossmodal semantic interaction is not constrained by the need for the strict temporal coincidence of the constituent auditory and visual stimuli. We therefore suggest that audiovisual semantic interactions likely occur in a short-term buffer which rapidly accesses, and temporarily retains, the semantic representations of multisensory stimuli in order to form a coherent multisensory object representation. These results are explained in terms of Potter's (1993) notion of conceptual short-term memory. (c) 2009 Elsevier B.V. All rights reserved.
引用
收藏
页码:389 / 404
页数:16
相关论文
共 138 条
[71]   Grabbing your ear: Rapid auditory-somatosensory multisensory interactions in low-level sensory cortices are not constrained by stimulus alignment [J].
Murray, MM ;
Molholm, S ;
Michel, CM ;
Heslenfeld, DJ ;
Ritter, W ;
Javitt, DC ;
Schroeder, CE ;
Foxe, JJ .
CEREBRAL CORTEX, 2005, 15 (07) :963-974
[72]   Exposure to asynchronous audiovisual speech extends the temporal window for audiovisual integration [J].
Navarra, J ;
Vatakis, A ;
Zampini, M ;
Soto-Faraco, S ;
Humphreys, W ;
Spence, C .
COGNITIVE BRAIN RESEARCH, 2005, 25 (02) :499-507
[73]   INTERSENSORY FACILITATION OF REACTION-TIME - ENERGY SUMMATION OR PREPARATION ENHANCEMENT [J].
NICKERSON, RS .
PSYCHOLOGICAL REVIEW, 1973, 80 (06) :489-509
[74]   Audiovisual temporal correspondence modulates human multisensory superior temporal sulcus plus primary sensory cortices [J].
Noesselt, Toemme ;
Rieger, Jochem W. ;
Schoenfeld, Mircea Ariel ;
Kanowski, Martin ;
Hinrichs, Hermann ;
Heinze, Hans-Jochen ;
Driver, Jon .
JOURNAL OF NEUROSCIENCE, 2007, 27 (42) :11431-11441
[75]   DATA-LIMITED AND RESOURCE-LIMITED PROCESSES [J].
NORMAN, DA ;
BOBROW, DG .
COGNITIVE PSYCHOLOGY, 1975, 7 (01) :44-64
[76]   Constrained formation of object representations [J].
O'Connor, KJ ;
Potter, MC .
PSYCHOLOGICAL SCIENCE, 2002, 13 (02) :106-111
[77]  
Olivetti BelardinelliM., 2004, Cognitive Processing, V5, P167, DOI [10.1007/s10339%E2%80%90004%E2%80%900024%E2%80%900, DOI 10.1007/S10339-004-0024-0, 10.1007/s10339-004-0024-0.]
[78]   EFFECTS OF CONTEXTUAL SCENES ON IDENTIFICATION OF OBJECTS [J].
PALMER, SE .
MEMORY & COGNITION, 1975, 3 (05) :519-526
[79]   'When Birds of a Feather Flock Together': Synesthetic Correspondences Modulate Audiovisual Integration in Non-Synesthetes [J].
Parise, Cesare Valerio ;
Spence, Charles .
PLOS ONE, 2009, 4 (05)
[80]   Graded modality-specific specialisation in semantics: A computational account of optic aphasia [J].
Plaut, DC .
COGNITIVE NEUROPSYCHOLOGY, 2002, 19 (07) :603-639