We introduce a new method for hierarchical reinforcement learning. Highlevel policies automatically discover subgoals; low-level policies learn to specialize on different subgoals. Subgoals are represented as desired abstract observations which cluster raw input data. High-level value functions cover the state space at a coarse level; low-level value functions cover only parts of the state space at a fine-grained level. Experiments show that this method outperforms several flat reinforcement learning methods in a deterministic task and in a stochastic task.

this paper, the progress of this development is reviewed and analysed in detail. In order to structure the survey and to evaluate the techniques, a taxonomy, specifically designed for this purpose, has been developed. Moreover, important open research issues are identified, that, if addressed properly, possibly can give the field a significant push forward

Abstract not found

It is difficult to apply traditional reinforcement learning algorithms to robots, due to problems with large and continuous domains, partial observability, and limited numbers of learning experiences. This paper deals with these problems by combining: 1. reinforcement learning with memory, implemented using an LSTM recurrent neural network whose inputs are discrete events extracted from raw inputs; 2. online exploration and offline policy learning. An experiment with a real robot demonstrates the methodology's feasibility.

This paper describes how learning tasks in partially observable mobile robot domains can be solved by combining reinforcement learning with an unsupervised learning \event extraction" mechanism, called ARAVQ. ARAVQ transforms the robot's continuous, noisy, high-dimensional sensory input stream into a compact sequence of high-level events. The resulting hierarchical control system uses an LSTM recurrent neural network as the reinforcement learning component, which learns high-level actions in response to the history of high-level events. The highlevel actions select low-level behaviors which take care of real-time motor control. Illustrative experiments based on a Khepera mobile robot simulator are presented.

In autonomous agent systems, internal memory can be an important element to overcome the limitations of purely reactive agent behaviour. This paper presents an analysis of memory requirements for T-maze tasks well known as the road sign problem. In these tasks a robot agent should make a decision of turning left or right at the T-junction in the approach corridor, depending on a history of perceptions. The robot agent in simulation can sense the light intensity influenced by light lamps placed on the bank of the wall. We apply the evolutionary multiobjective optimization approach to finite state controllers with two objectives, behaviour performance and memory size. Then the internal memory is quantified by counting internal states needed for the T-maze tasks in noisy environments. In particular, we focused on the influence of noise on internal memory and behaviour performance, and it is shown that state machines with variable thresholds can improve the performance with a hysteresis effect to filter out noise. This paper also provides an analysis of noise effect on perceptions and its relevance on performance degradation in state machines. keywords: T-maze, delayed response task, evolutionary robotics, finite state machines, evolutionary multiobjective optimization, internal memory 1 1

We show how a simple layered system can selforganize into a set of distinct states and qualitatively different behaviors as a result of learning a robotic delayed response task. Our approach is based on an architecture where higher levels are able to dynamically modulate the lower reactive mapping when needed.

Abstract — The road sign problem is tackled in this work with Reservoir Computing (RC) networks. These networks are made of a fixed recurrent neural network where only a readout layer is trained. In the road sign problem, an agent has to decide at some point in time which action to take given relevant information gathered in the past. We show that RC can handle simple and complex T-maze tasks (which are a subdomain of the road sign problem). Keywords — Reservoir Computing, road sign problem, T-maze, long-term memory. 1

We present a design for general layered control systems capable of learning delayed response tasks with arbitrarily long delays. The approach is based on extracting events from the time-step based input stream and reacting to these events using top-down modulation. This modulation can bias the outputs in some direction and lets the system focus on a subset of the available sensory array. To avoid information distortion as inputs passes through layers, all layers are connected directly to the input, like in the Subsumption Architecture. However, here higher layers also have access to an additional information channel from lower layers, specifying whether the particular input seems relevant. Thereby higher layers can lay out more long-term plans without distractions, only being interrupted or notified when important input conditions arise.

In this work we tackle the road sign problem with Reservoir Computing (RC) networks. The T-maze task (a particular form of the road sign problem) consists of a robot in a T-shaped environment that must reach the correct goal (left or right arm of the T-maze) depending on a previously received input sign. It is a control task in which the delay period between the sign received and the required response (e.g., turn right or left) is a crucial factor. Delayed response tasks like this one form a temporal problem that can be handled very well by RC networks. Reservoir Computing is a biologically plausible technique which overcomes the problems of previous algorithms such as Backpropagation Through Time- which exhibits slow (or non-) convergence on training. RC is a new concept that includes a fast and efficient training algorithm. We show that this simple approach can solve the T-maze task efficiently.