Bridging Brain, Mind, and Behavior: 2006 Research Awards

Columbia University, New York, New York
Principal Investigator: Charles D. Salzman, $450,000 over three years.
Learning about value in a complex world: neurophysiology and theory



Imagine that you are playing blackjack, and you have just been dealt the king of spades. Can you tell what your emotional reaction would be? Of course you can't! Not without knowing the context – the cards you were dealt previously. The emotional "value" of the king could range from positive (if it completes your hand to make 21) to negative (if it makes a total greater than 21). To generate the appropriate emotional response to a stimulus, animals and people need to learn two things: the possible meanings a stimulus can have – can it lead to a big reward? a small reward? a punishment? – and which meaning a stimulus will actually have given the behavioral context. How does this kind of emotional learning take place? Our research plan, a proposed collaboration between experimental and theoretical neuroscientists, aims to find out the neural basis behind a simplified form of this kind of learning: context-dependent reinforcement learning.

Many of our emotional responses (which can be cognitive, physiological, or behavioral) occur in relation to sensory stimuli that predict impending rewards or punishments. For example, when we see appetizing foods, we learn to take actions to acquire them. We also learn to avoid or defend against stimuli that are punishing or pose threats to us. Since Pavlov, conditioning techniques have been used to investigate how humans and animals learn to anticipate reinforcement based on sensory stimuli – a process called reinforcement learning. In classical (Pavlovian) conditioning, sensory stimuli are mapped in a one-to-one fashion with a reinforcement outcome (reward or punishment). However, our ability to predict reinforcement clearly extends to much more complex situations. Case in point: we routinely learn that individual stimuli can have different values depending upon the moment-to-moment context in which a stimulus appears (as in the blackjack example above). In this research proposal, we propose combining neurophysiological experiments in alert, behaving monkeys with theoretical work to examine how such context-dependent reinforcement learning occurs in the brain. We study monkeys because their brains and behavior are similar to those of humans in many ways. In particular, monkeys, like humans, are highly visual animals; we both use visual-emotional associations to guide much of our motivated behavior.

How can scientists study such a nebulous concept as emotion? In the last 25 years, neuroscientists like us have begun to tackle the neural underpinnings of emotion by defining emotion "operationally." This means that we attempt to dissect out a certain aspect of emotion – fear, for example – and find behavioral or physiological ways to measure it in animals – say, avoidance of specific objects, or elevated heart rate. At the same time, we can measure or manipulate brain activity in order to understand the neural processes underlying that aspect of emotion. This approach has led us to realize that studying the neural basis of emotion is no different from studying the neural basis of other cognitive operations, such as perception, memory, decision making, or the planning of movements. To investigate any of these, we combine techniques from experimental psychology and neuroscience to understand how neural computations produce cognitive and emotional behavior. Furthermore, as our understanding grows, it becomes critical to develop quantitative theoretical models that describe how neural circuits mediate higher brain processes. Because the development of models leads to predictions which can then be tested experimentally, it is through continual interactions between experimental and theoretical work that a more detailed understanding of brain function will emerge.

The proposed work builds on recent findings in our lab, where we focus on the amygdala – a brain area that is known to be essential for many kinds of emotional behavior. We have discovered that neurons in the monkey amygdala change their activity as monkeys learn that visual stimuli are associated with rewards and punishments¹. Some neurons responded more strongly when an image predicted a reward, thereby encoding the “positive value” of visual stimuli. Other neurons fired more strongly when an image predicted a punishment, encoding the “negative value” of visual stimuli. In these experiments, we manipulated the value of images across blocks of trials by changing their association with rewards and punishments. Strikingly, individual amygdala neurons changed their activity at about the same time (on average) that monkeys learned the new values of images. These data indicate that the amygdala contains a neural "representation" of visual stimulus value, and that monkeys could have used this representation to guide their behavior.

Our discovery raises a host of new questions: Through what mechanisms does the representation of value in the amygdala change? Is the representation adjusted by gradually changing the strengths of inputs onto neurons in the amygdala? Or is the adjustment driven by a "switching mechanism" that facilitates more rapid changes in inputs? Or could both sorts of mechanisms exist in the amygdala? We hope that studying context-dependent reinforcement learning – which we will do by combining neurophysiological experiments and computational modeling – will help us to answer these and other related questions. Specifically, we propose to investigate amygdala neural responses while monkeys learn how multiple contextual cues manipulate the value of visual stimuli on a trial-by-trial basis. In parallel, we will develop computational models that extend recent efforts by our collaborator, Stefano Fusi, aimed at understanding the theoretical basis of learning and memory.

We plan to record neural responses in the amygdala while monkeys perform a context-dependent reinforcement learning task. In this task, the value of an image depends upon which of two possible contextual cues appears before the image on each trial. In this way, the image is like the king of spades in the blackjack example: it can have a positive or negative value depending upon the context. This experimental design allows us to manipulate the value of a particular image on a trial-by-trial basis. We can then ask how amygdala neural activity tracks image value as monkeys learn about the contextual cues. If neural activity in the amygdala can switch rapidly as image value changes from trial to trial, it would suggest that a context-dependent mechanism facilitates switching between two representations of value for the same image.

In parallel, we will develop a realistic model of a neural circuit that can simulate such context-dependent reinforcement learning. The model will contain two basic learning mechanisms. The first mechanism will perform simple mappings of images onto reinforcement outcomes, as occurs during classical conditioning. The second mechanism will implement the storage of two or more contexts simultaneously in the neural circuit, enabling contextual cues to activate a specific context representation. Together, these two mechanisms will allow contextual cues to dictate image values on a trial-by-trial basis. A unique feature of our model (as currently conceived) is that the strengths of connections between "neurons" can be modified to support learning, but some connections can be changed quickly and some can only be changed more slowly. This will allow the model to "learn" and "remember" information on both short and long timescales, resulting in a model that is both reliable and flexible.

The long-term goal of developing this model is to predict the behavior of monkeys, as well as neural activity in the amygdala, over a range of emotional learning situations. The model will be used to guide future data analyses and experimental manipulations; in return, these will allow us to test and refine the model so as to better describe both neural activity and behavior. In particular, we would like to test specific components of the model, such as the important idea that the connections between neurons can exist in different “states” where they integrate information over different timescales.

The work that we propose has long-range implications. The ability to form representations of stimulus value that can be flexibly activated, depending upon context, is a fundamental aspect of human emotion and cognition. It's important to note that context can be established by your external environment (e.g. the cards in your hand) or your internal state of mind (e.g. your perceived level of desire to win the hand). By combining computational and neurophysiological approaches, we hope to shed light on the mechanisms underlying a complex form of reinforcement learning in which contexts dictate how sensory stimuli predict reinforcement. Our work might also lead to more general principles that can advance our understanding of other cognitive functions for which context is important. Finally, we know that when the neural circuitry that underlies emotional processing goes awry, it can result in psychiatric problems ranging from post-traumatic stress disorder to depression. Understanding this circuitry in a more detailed fashion – as we hope to do – promises to point the way to finding out the causal factors, and ultimately more effective treatments, for disorders of emotion.

1. Paton, J., Belova, MA, Morrison, SE, Salzman, CD. The primate amygdala represents the positive and negative value of visual stimuli during learning. Nature 439, 865-870 (2006).