An integrative computational account of language and locomotion

Word meanings across languages carve up human experience in ways that are believed to be optimized for efficient communication. This theory of language cognition has been tested on adjectives (e.g., colors), nouns (e.g., containers), and function words (e.g., pronouns) primarily through static stimuli. However, language cognition in the real world involves talking about dynamic processes that are perceived through multi-sensory integration, embodied, and acted upon. Here, we propose to study such dynamic language cognition through the lens of verb meanings: Do verb meanings across languages represent body movements in ways that allow efficient communication about these dynamic processes in the world? We aim to understand the cognitive constraints and computational principles that shape the ways in which humans use language to represent movements. More specifically, we aim to study how native speakers of different languages use verbs to describe movements, and how they comprehend verb meanings by demonstrating movements.

These questions pose two open challenges. First, in contrast to other semantic domains, which have often been grounded in static elements of the environment, verbs refer to dynamical processes that are a result of multi-sensory perception-action loops which are much harder to characterize. Second, in order to test theories about language for movement, we need large-scale cross-linguistic production and comprehension data which does not currently exist.

Here, we propose to address these open challenges by (1) using state-of-the-art computational models of motor control to develop a locomotion meaning space and generate from it a large dynamic stimulus set; (2) collecting large-scale cross-linguistic verb production and comprehension data with respect to our stimulus set; (3) supplementing theses data with in-lab experiment to measure the motion and forces as individuals demonstrate their comprehension of locomotion verbs; and (4) generating theory-driven predictions by integrating the computational locomotion models with the computational framework of efficiency in language, and then testing these predictions on our newly collected data.

Our proposed work will help develop a theory of language cognition that is more naturalistic by seeking to encompass meanings that represent the dynamic and multi-sensory processes involved in locomotion. This proposal builds on the team's complementing expertise in computational psycholinguistics and the cognitive science of language (PI Levy), information theory and cross-language semantic variation (Dr. Zaslavsky), and computational modeling of human motor control (Dr. Seethapathi), in addition to the expertise of all collaborators in contemporary machine learning methods.