Actor-critic models of the basal ganglia: new anatomical and computational perspectives

被引:296
作者
Joel, D [1 ]
Niv, Y
Ruppin, E
机构
[1] Tel Aviv Univ, Dept Psychol, IL-69978 Tel Aviv, Israel
[2] Tel Aviv Univ, Sch Med, IL-69978 Tel Aviv, Israel
[3] Tel Aviv Univ, Sch Math Sci, IL-69978 Tel Aviv, Israel
关键词
basal ganglia; dopamine; reinforcement learning; actor-critic; dimensionality reduction; evolutionary computation; behavioral switching; striosomes/patches;
D O I
10.1016/S0893-6080(02)00047-3
中图分类号
TP18 [人工智能理论];
学科分类号
081104 ; 0812 ; 0835 ; 1405 ;
摘要
A large number of computational models of information processing in the basal ganglia have been developed in recent years. Prominent in these are actor-critic models of basal ganglia functioning, which build on the strong resemblance between dopamine neuron activity and the temporal difference prediction error signal in the critic, and between dopamine-dependent long-term synaptic plasticity in the striatum and learning guided by a prediction error signal in the actor. We selectively review several actor-critic models of the basal ganglia with an emphasis on two important aspects: the way in which models of the critic reproduce the temporal dynamics of dopamine firing, and the extent to which models of the actor take into account known basal ganglia anatomy and physiology. To complement the efforts to relate basal ganglia mechanisms to reinforcement learning (RL), we introduce an alternative approach to modeling a critic network, which uses Evolutionary Computation techniques to 'evolve' an optimal RL mechanism, and relate the evolved mechanism to the basic model of the critic. We conclude our discussion of models of the critic by a critical discussion of the anatomical plausibility of implementations of a critic in basal ganglia circuitry, and conclude that such implementations build on assumptions that are inconsistent with the known anatomy of the basal ganglia. We return to the actor component of the actor-critic model, which is usually modeled at the striatal level with very little detail. We describe an alternative model of the basal ganglia which takes into account several important, and previously neglected, anatomical and physiological characteristics of basal ganglia-thalamocortical connectivity and suggests that the basal ganglia performs reinforcement-biased dimensionality reduction of cortical inputs. We further suggest that since such selective encoding may bias the representation at the level of the frontal cortex towards the selection of rewarded plans and actions, the reinforcement-driven dimensionality reduction framework may serve as a basis for basal ganglia actor models. We conclude with a short discussion of the dual role of the dopamine signal in RL and in behavioral switching. (C) 2002 Elsevier Science Ltd. All rights reserved.
引用
收藏
页码:535 / 547
页数:13
相关论文
共 82 条
[1]   FUNCTIONAL ARCHITECTURE OF BASAL GANGLIA CIRCUITS - NEURAL SUBSTRATES OF PARALLEL PROCESSING [J].
ALEXANDER, GE ;
CRUTCHER, MD .
TRENDS IN NEUROSCIENCES, 1990, 13 (07) :266-271
[2]  
[Anonymous], 1995, MODELS INFORM PROCES
[3]   3-D tracing of biocytin-labelled pallido-thalamic axons in the monkey [J].
ArecchiBouchhioua, P ;
Yelnik, J ;
Francois, C ;
Percheron, G ;
Tande, D .
NEUROREPORT, 1996, 7 (05) :981-984
[4]   Is heterosynaptic modulation essential for stabilizing Hebbian plasticity and memory? [J].
Bailey, CH ;
Giustetto, M ;
Huang, YY ;
Hawkins, RD ;
Kandel, ER .
NATURE REVIEWS NEUROSCIENCE, 2000, 1 (01) :11-20
[5]  
Bar-Gad Izhar, 2000, Journal of Basic and Clinical Physiology and Pharmacology, V11, P305
[6]  
Barto AG., 1995, Models of information processing in the basal ganglia, P215
[7]   TOPOGRAPHICAL ORGANIZATION AND RELATIONSHIP WITH VENTRAL STRIATAL COMPARTMENTS OF PREFRONTAL CORTICOSTRIATAL PROJECTIONS IN THE RAT [J].
BERENDSE, HW ;
GALISDEGRAAF, Y ;
GROENEWEGEN, HJ .
JOURNAL OF COMPARATIVE NEUROLOGY, 1992, 316 (03) :314-347
[8]   A computational model of how the basal ganglia produce sequences [J].
Berns, GS ;
Sejnowski, TJ .
JOURNAL OF COGNITIVE NEUROSCIENCE, 1998, 10 (01) :108-121
[9]   What is the role of dopamine in reward: hedonic impact, reward learning, or incentive salience? [J].
Berridge, KC ;
Robinson, TE .
BRAIN RESEARCH REVIEWS, 1998, 28 (03) :309-369
[10]  
Brown J, 1999, J NEUROSCI, V19, P10502