Abstract

Humans and other animals often engage in activities for their own sakes rather than as steps toward solving practical problems. Psychologists call these intrinsically motivated behaviors. What we learn during intrinsically motivated behavior is essential for our development as competent autonomous entities able to efficiently solve a wide range of practical problems as they arise. In this paper we present initial results from a computational study of intrinsically motivated learning aimed at allowing artificial agents to construct and extend hierarchies of reusable skills that are needed for competent autonomy. At the core of the model are recent theoretical and algorithmic advances in computational reinforcement learning, specifically, new concepts related to skills and new learning algorithms for learning with skill hierarchies. 1

Citations

2827	Reinforcement Learning: An Introduction - Sutton, Barto - 1998
342	Between MDPS and semi-MDPS: a framework for temporal abstraction in reinforcement learning - Sutton, Precup, et al. - 1999
324	PR: A neural substrate of prediction and reward - Schultz, Dayan, et al. - 1997
244	The Origins of Intelligence in Children - Piaget - 1963
222	Motivated reinforcement learning - Dayan - 2002
182	Predictive reward signal of dopamine neurons - Schultz - 1998
150	A framework for mesencephalic dopamine systems based on predictive Hebbian learning - PR, Dayan, et al. - 1996
134	Policy invariance under reward transformations: Theory and applications to reward shaping - Ng, Harada, et al. - 1999
119	Recent advances in hierarchical reinforcement learning. Discrete Event Dynamic Systems - Barto, Mahadevan - 2003
93	A model of how the basal ganglia generate and use neural signals that predict reinforcement - Houk, Adams, et al. - 1995
68	Motivation reconsidered: The concept of competence - White - 1959
65	Discovering hierarchy in reinforcement learning with HEXQ - Hengst
59	Intrinsically motivated reinforcement learning - Singh, Barto, et al. - 2005
57	A possibility for implementing curiosity and boredom in model-building neural controllers - Schmidhuber - 1991
46	Novelty and reinforcement learning in the value system of developmental robots - Huang, Weng - 2002
42	Autonomous discovery of temporal abstractions from interaction with an environment - McGovern - 2002
31	Exploration Bonuses and Dual Control - Dayan, Sejnowski - 1996
29	Value-dependent selection in the brain: simulation in a synthetic neural model. Neuroscience 59 - KJ, Tononi, et al. - 1994
29	Hochreiter S: Reinforcement Driven Information Acquisition in Non-Deterministic Environments - Schmidhuber, Storck - 1995
28	Integrated modeling and control based on reinforcement learning and dynamic programming - Sutton - 1991
26	Dopamine: Generalization and bonuses - Kakade, Dayan - 2002
24	Motivational principles for visual know-how development - Kaplan, Oudeyer - 2003
19	Many-layered learning - Utgoff, Stracuzzi - 2002
10	Drug addiction as dopamine-dependent associative learning disorder - Chiara - 1999
9	Dopamine bonuses - Kakade, Dayan - 2001
9	Curious design agents and artificial creativity: A synthetic approach to the study of creative behavior. Doctoral dissertation - Saunders - 2002
7	Burst activity of ventral tegmental dopamine neurons is elicited by sensory stimuli in the awake cat - Horvitz, Stewart, et al. - 1997
6	Intrinsic reinforcing properties of putatively neutral stimuli in an instrumental two-lever discrimination task - Reed, Mitchell, et al. - 1996
4	Learning hierarchical control structure from multiple tasks and changing environments. From Animals to Animats 5 - Digney - 1998
1	Autonomous mental develoopment by robots and animals - Wang, McClelland, et al.