McClure et al (2003) A computational substrate for incentive salience

McClure, S. M., Daw, N. D., & Montague, P. R. (2003). A computational substrate for incentive salience. TRENDS in Neuroscience, 26, 423-428.

From abstract:
Theories of dopamine function have two families of models that use dopamine neuron firing rates as a prediction error signal: (1) learning to predict future rewarding events, and (2) biasing action choice.
Pharmacological inhibition or lesion of dopaminergic neurons diminishes animals' abilities to motivate behaviors directed at acquiring rewards. These indicate that dopamine release may encode a measure of incentive value of a contemplated behavioral act. The most complete psychological idea that captures this notion frame the dopamine signal as carrying "incentive salience." This paper demonstrates that both functions (prediction and action choice bias) can be captured in a single computational model.

o   Incentive salience (McClure et al, 2003; also Berridge & Robinson, 1998; Salamone & Correa, 2002)

§  Dopamine doesn’t change the interval value of the reward, but is necessary to initiate action to act on that valuation.

§  Dopamine release assigns incentive value to objects or behavioral acts. These assigned values are then available to be used by an action selection system that makes more valuable actions more likely.

§  Accordingly, antagonism of dopamine receptor function doesn’t influence the assignment of value, but does inhibit the use of these values in choosing actions.

§  Does not affect motivation for the reward (rats with antagonist will drink sucrose just s vigorously if placed in front of it, but are way slower to move down a corridor to get it.

o   Prediction error

§  Incentive salience theory starts from once a goal is already selected. But how does an organism choose actions to obtain food, sex, and other natural rewards?

§  TD learning algorithm. Dopamine neuron spiking encodes a D prediction error signal.

§  In the TD model, dopamine serves two purposes:

·      Used as a learning signal for value function V, therefore hypothesized to be required for learning to predict future rewards.

·      Dopamine biases action selection toward situations predictive of reward.

§  Dopamine might also have direct, immediate influence on action choice:

·      E.g., as prediction error reports whether payoff is better or worse than expected, it can indicate whether to continue a course of action or try something else.

o   Proposal: the concept of incentive salience is the expected future reward. Additionally, the role of dopamine in learning to attribute such expectations to situations that are predictive of reward and in biasing action selection serve as the formal counterpart for Berridge and Robinson’s theories on incentive salience.

§  They have a simple model in which actions are selected using incentive salience, and learned using prediction error. A factor that decreases the error prediction signal slows both learning and action selection.

The paper also gives a very good review/description of the TD learning algorithm, and presents this whole thing mathematically.