Event Cognition in the Wild

Events are the fundamental building blocks of human experience, and they are the things we remember when we look back at our lives. Over the last 20 years, the scientific field of event cognition has seen dramatic growth in explaining how the mind and brain deal with brief, simple everyday events. A major catalyst has been the ability to present complex video, text and VR stimuli while measuring behavior and brain activity. We have already seen payoffs in clinical science and information technology.

However, event cognition research is still largely locked in the lab, where we can only study "nano-events" such as words, pictures, and sentences, or at best "micro-events" such as short movies or curated interactions. To make the next qualitative jump forward will require getting out of the lab and developing techniques that allow researchers to record what people actually do and how they think about it over ecologically meaningful temporal extents. That is what we propose to do here.

The rise of smartphones, online services, and the internet of things make this the right time to tackle human event comprehension and memory at the scale of our lived experience. In this proposal, we will take advantage of these developments to study event cognition in the wild. Over the last 10 years, Professor Simon Dennis and his collaborators have been developing an infrastructure for helping people record cues as they live their lives that can help them personally to enrich their personal memories, and also form the basis for new discoveries about memory (see www.unforgettable.me). Unforgettable is capable of collecting over 600 sources of information, including time, location (GPS coordinates and place names), people (generated from emails, SMS and phone calls), activities (accelerometry 8-15 times a second continuously plus labels from activity recognition classifiers), images, sounds, language (email and SMSs), weather, and self-reported emotions. Unforgettable Research Services has a pool of 2700 participants - some of whom have been collecting these sources continuously for the last four years. In this project, Zacks, Dennis, and their team will use these existing data along with new data we will collect to build and stress-test computational models of event comprehension and memory that embrace the complexity and scale of real life.

This project is uniquely suited to the JMSF call. It embraces complexity by taking on the full range of everyday activity situated in real work (and play) context. It recognizes the dynamic nature of action by explicitly tracking the multidimensional interactivity of naturally occurring activity. It unites previously separate domains of activity by merging data from live behavior (sound and picture recordings, GPS) with data from the increasingly essential virtual dimensions of our lives (email, SMS). By doing so, we will unite the traditionally separated domains of psychology, ethology, big data, and human-technology interaction.

The events that matter in our lives are too big and too complicated to fit within a laboratory session. Now is time to go study events in the wild.