Synchronization of hand movement and fundamental frequency in a sample of spoken Spanish

Abstract

This study based on a multimodal approach to communication presents an analysis of the hand movement in relation to voice variation. The corpus consisted of a set of speech samples from a video-recorded interview conducted by a Venezuelan female TV-show host. The speech signal segmentation in syllables was based on an oscillogram and on the automatic detection of the fundamental frequency. The voice analysis was performed using a CSL of the Kay Elemetrics Corporation, and the analysis of the gestural movement was based on the observation of the videorecorded images. According to the simultaneous analysis of right and left hand orientation and the fundamental frequency, three different degrees of relationship were observed: a) a close relationship between the f0 patterns and the hand movements associated with the presence of either emphatic speech, parenthetic comments, or iconic gestures; b) a mild relationship between the f0 patterns and the hand movements, and c) absence of relationship. These findings suggest that the association between vocal and manual gestures is produced beyond the lexical level.