Characterizing infants’ everyday motor and object experiences through computer vision analysis of caregiver-captured video surveys

Although theories of infant development stress the importance of everyday experiences for honing infants' motor skills, object knowledge, and language, the structure and content of infants' day-to-day lives is notoriously difficult to measure. Ideally, such measurements would simultaneously: 1) record rich data about multimodal experiences, 2) densely sample within a day and across days to capture variability in activities (e.g., errands, meals, play) over multiple timescales, 3) facilitate testing large, representative participant samples. Yet, no existing method satisfies all three criteria. Video observations in the home produce rich data, but are limited to a brief slice of time within a single day (usually while infants play). Day-long recording methods (wearable motion sensors, audio recorders) require specialized equipment, so are difficult to apply at scale. Daily diaries and surveys can broadly sample time and scale easily, but produce limited text response data.

In this project, we will innovate a new method that employs caregiver-captured video to glean new insights about infants' everyday learning opportunities. We will apply the method to test new questions about how learning to walk alters infants' daily motor and object experiences. Leveraging the ubiquity of camera-enabled smartphones, we will prompt caregivers to take 5-s videos of their infants each hour of each day for two weeks (168 samples). We will apply a computer vision algorithm and train a machine learning classifier to automatically detect infant body position (upright, sitting, supine, and prone) from video to reduce the need for human coding of a large dataset.

In Year 1, we will validate the new method in 13-month-old infants. Hand-coded body position annotations will train and test the machine learning model. The resulting body position categories will reveal how much time infants spend upright, prone, supine, and sitting in daily life, and how body position frequency differs between infants who have begun walking to those who have not. Crucially, time sampling within and across days will allow us to document day-to-day variability in body position.

In Years 2 and 3, we will apply the method at scale to investigate how everyday experiences might mediate the link between walking and word learning via object experiences. Although recent work shows that infants who can walk have larger vocabularies compared to infants who cannot, it is unknown what aspect of walking facilitates this effect. We hypothesize that walking allows infants to acquire objects of interest and interact with them across a more diverse range of physical and activity contexts. To test this prediction, we will conduct a longitudinal video-survey study to measure infants' body position and object holding across the transition to walking.

Openly sharing the method will provide a new tool for researchers to gather naturalistic data about a wide range of daily experiences. Sharing the corpus of caregiver-captured video will allow secondary data analysis and provide a benchmark dataset for testing new computer vision classifiers.