Talk by Andy Lücking at the 1st International Workshop on AI Aspects of Reasoning, Information, and Memory, 11 – 14 September, 2016

Talk by Andy Lücking at the 1st International Workshop on AI Aspects of Reasoning, Information, and Memory

11 – 14 September, 2016, Gdansk, Poland

Modeling Co-Verbal Gesture Perception in Type Theory with Records

Abstract
In natural language \textit{face to face} communication interlocutors exploit manifold non-verbal information resources, most notably hand and arm movements, i.e. gestures. In this paper, a type-theoretical approach using Type Theory with Records is introduced which accounts for iconic gestures within an information state update semantics. Iconic gestures are semantically exploited in two steps: firstly, their kinetic representations are mapped onto vector sequence representations from vector space semantics, modeling a perceptual gesture classification; secondly, these vectorial representation are linked to linguistic predicates giving rise to a computational account to semantic-kinematic interfaces. Each of the steps involves reasoning processes, which are made explicit. The resulting framework shows how various resources have to be integrated in the update mechanism in order to deal with apparently simple multimodal utterances.