M.A. Wiering and J. Schmidhuber. Solving POMDPs with Levin search and EIRA. In L. Saitta, editor, Machine Learning: Proceedings of the Thirteenth International Conference, pages 534-542. Morgan Kaufmann Publishers, San Francisco, CA, 1996.  Home/Search   Document Details and Download   Summary   Related Articles   Check

....100 papers on diverse topics including fine arts [96] and the nature of surprises [97] Apparently he even founded a religion [94] Most of his articles, however, are about machines that learn from experience. I have started to compile an incomplete list of references to work by him and his lab [117, 116, 39, 50, 40, 42, 43, 41, 52, 49, 56, 44, 54, 47, 48, 51, 53, 57, 46, 68, 45, 55, 69, 64, 65, 59, 66, 58, 67, 60, 63, 61, 73, 71, 79, 70, 74, 62, 72, 75, 78, 82, 80, 76, 81, 77, 84, 89, 88, 94, 87, 85, 96, 83, 100, 86, 90, 99, 91, 93, 105, 119, 95, 92, 97, 120, 118, 98, 125, 130, 129, 126, 128, 124, 123, 122, 131, 127, 35, 34, 36, 38, 32, 33, 37, 27, 28, 25, 24, 22, 23, 15, 9, 21, 10, 16, 26, 17, 18, 6, 7, 8, 13, 11, 20, 19, 14, 12, 115, 114, 121, 30, 106, 108, 107, 29, 31, 109, 110, 111, 112, 113, 5, 101, 103, 104, 4, 3, 2, 1, 102]. Hopefully I ll be able to add missing entries soon. Future work will concentrate on categorizing related papers and establishing common threads. ....

M.A. Wiering and J. Schmidhuber. Solving POMDPs with Levin search and EIRA. In L. Saitta, editor, Machine Learning: Proceedings of the Thirteenth International Conference, pages 534--542. Morgan Kaufmann Publishers, San Francisco, CA, 1996.

No context found.

M.A. Wiering and J. Schmidhuber. Solving POMDPs with Levin search and EIRA. In L. Saitta, editor, Machine Learning: Proceedings of the Thirteenth International Conference, pages 534-542. Morgan Kaufmann Publishers, San Francisco, CA, 1996.

No context found.

M.A. Wiering and J. Schmidhuber. Solving POMDPs with Levin search and EIRA. In L. Saitta, editor, Machine Learning: Proceedings of the Thirteenth International Conference, pages 534-542. Morgan Kaufmann Publishers, San Francisco, CA, 1996.

No context found.

M.A. Wiering and J. Schmidhuber. Solving POMDPs with Levin search and EIRA. In L. Saitta, editor, Machine Learning: Proceedings of the Thirteenth International Conference, pages 534-542. Morgan Kaufmann Publishers, San Francisco, CA, 1996.

....of alternative programs decrease appropriately. Subsequent Lsearches for new problems then use the adjusted P , etc. A nonuniversal variant of this approach was able to solve reinforcement learning (RL) tasks [27] in partially observable environments unsolvable by traditional RL algorithms [71, 57]. Each Lsearch invoked by Als is optimal with respect to the most recent adjustment of P . On the other hand, the modi cations of P themselves are not necessarily optimal. Recent work discussed in the next section overcomes this drawback in a principled way. 9 Optimal Ordered Problem Solver ....

M.A. Wiering and J. Schmidhuber. Solving POMDPs with Levin search and EIRA. In L. Saitta, editor, Machine Learning: Proceedings of the Thirteenth International Conference, pages 534-542. Morgan Kaufmann Publishers, San Francisco, CA, 1996.

.... time is still spent on T s maximal value (ignoring hardware speci c overhead for parallelization and nonessential speed ups due to halting programs if there are any) Nonbinary, nonuniversal variants of Osearch were used to solve machine learning toy problems unsolvable by traditional methods [58, 47]. Probabilistic alternatives based on probabilistically chosen maximal program runtimes in Speed Prior style [41, 45] also outperformed traditional methods on certain toy problems [39, 40] 2.4 Incremental Search Since Newell Simon s early attempts at building a General Problem Solver [32, ....

....of alternative programs decrease appropriately. Subsequent Osearches for new problems then use the adjusted P , etc. A nonuniversal variant of this approach was able to solve reinforcement learning (RL) tasks [20] in partially observable environments unsolvable by traditional RL algorithms [58, 47]. Each Osearch invoked by Als is bias optimal with respect to the most recent adjustment of P . On the other hand, the rather arbitrary P modi cations themselves are not necessarily optimal. They may quickly distort the initial bias, and provoke a loss of nearbias optimality with respect to the ....

M.A. Wiering and J. Schmidhuber. Solving POMDPs with Levin search and EIRA. In L. Saitta, editor, Machine Learning: Proceedings of the Thirteenth International Conference, pages 534-542. Morgan Kaufmann Publishers, San Francisco, CA, 1996.

....of the environment. While DPRL is essentially limited to learning reactive policies mapping current inputs to output actions, EC in principle can be applied to search spaces whose elements are general algorithms or programs with time varying variables that can be used for memory purposes [52, 46, 32, 51, 27]. EC Advantage 3: Straight forward Hierarchical Credit Assignment. There has been a lot of recent work on hierarchical DPRL. Some researchers address the case where an external teacher provides intermediate subgoals and or prewired macro actions consisting of sequences of lower level actions [21, ....

....framework, however, hierarchical credit assignment via frequently used, automatically generated subprograms becomes trivial in principle. For instance, suppose policies are programs built from a general programming language that permits parameterized conditional jumps to arbitrary code addresses [6, 22, 51, 37, 35]. EC will simply keep successful hierarchical policies that partially reuse code (subprograms) via appropriate jumps. Again, partial observability is not an issue. EC Advantage 4: Non Hierarchical Abstract Credit Assignment. Hierarchical learning of macros and reusable subprograms is of interest ....

M.A. Wiering and J. Schmidhuber. Solving POMDPs with Levin search and EIRA. In L. Saitta, editor, Machine Learning: Proceedings of the Thirteenth International Conference, pages 534--542. Morgan Kaufmann Publishers, San Francisco, CA, 1996.

....and memorize important past events in complex, partially observable settings, such as in the introductory example above. Several recent, non standard RL approaches in principle are able to deal with partially observable environments and can learn to memorize certain types of relevant events [7, 8, 6, 3, 9, 10, 13, 18, 17, 14]. None of them, however, represents a satisfactory solution to the general problem of learning in worlds described by partially observable Markov Decision Processes This work was supported by SNF grants 21 55409.98 and 2000 61847.00 (POMDPs) The approaches are either ad hoc, or work in ....

M.A. Wiering and J. Schmidhuber. Solving POMDPs with Levin search and EIRA. In L. Saitta, editor, Machine Learning: Proceedings of the Thirteenth International Conference, pages 534-542. Morgan Kaufmann Publishers, San Francisco, CA, 1996.

....scanning heads across large sections of its internal tapes. This may consume more time than necessary. To overcome potential slow downs of this kind, and to optimize the TM specific constant factor, we will slightly modify an optimal search algorithm called Levin search [36, 38, 1, 39] see [52, 70, 54] for the first practical applications we are aware of) Essentially, we will strip Levin search of its search aspects and apply it to possibly infinite objects. This leads to the most efficient (up to a constant factor depending on the TM) algorithm for computing all computable bitstrings. FAST ....

M.A. Wiering and J. Schmidhuber. Solving POMDPs with Levin search and EIRA. In L. Saitta, editor, Machine Learning: Proceedings of the Thirteenth International Conference, pages 534--542. Morgan Kaufmann Publishers, San Francisco, CA, 1996.

....scanning heads across large sections of its internal tapes. This may consume more time than necessary. To overcome potential slow downs of this kind, and to optimize the TM specific constant factor, we will slightly modify an optimal search algorithm called Levin search [53, 55, 1, 56] see [73, 97, 76] for the first practical applications we are aware of) Essentially, we will strip Levin search of its search aspects and apply it to possibly infinite objects. This leads to the most efficient (up to a constant factor depending on the TM) algorithm for computing all computable bitstrings. FAST ....

M.A. Wiering and J. Schmidhuber. Solving POMDPs with Levin search and EIRA. In L. Saitta, editor, Machine Learning: Proceedings of the Thirteenth International Conference, pages 534--542. Morgan Kaufmann Publishers, San Francisco, CA, 1996.

....single life reinforcement learning. To illustrate basic aspects of the principle, the remainder of this paper will also present a few preliminary ex15 periments. These few experiments, however, in no way represent a systematic experimental analysis, which is left for future papers (see also [47, 50]) The preliminary experiments in the current section demonstrate that the system from section 2 indeed can learn to compute SMSs leading to faster and faster reinforcement intake. The system uses low level problem specific instructions in addition to the 17 general, assembler like instructions ....

....effects of introducing different kinds of initial bias, or to compare the system to other learning systems with different initial bias (this is left for future work) Instead, this section s purpose is to illustrate typical aspects of the system s basic (bias independent) mode of operation. See [47, 50, 39] for additional, more complex experiments. 3.1 Writing Variable Sequences Task. The external environment consists of an array of 30 variables V 0 ; V 1 ; V 29 . The i th variable is denoted by V i . Its current contents are denoted by C i 2 [ GammaM axint; Maxint] Time is measured in ....

[Article contains additional citation context not shown here]

M. Wiering and J. Schmidhuber. Solving POMDPs with Levin search and EIRA. In L. Saitta, editor, Machine Learning: Proceedings of the Thirteenth International Conference. Morgan Kaufmann Publishers, San Francisco, CA, 1996. To appear.

....preferred over results with high Levin complexity. In an alternative implementation, the original universal search algorithm was used to systematically generate all solution candidates in order of their Levin complexities. See also Heil s MSc thesis at Technische Universitat Munchen (1995) and (Wiering and Schmidhuber, 1996)) Used primitives. The instruction numbers and the semantics of the primitives used in the experiments are listed below. An expression of the form address i denotes the value (interpreted as an address) found in the ith cell following the one containing the current instruction (indirect ....

....When speed is an issue, then we will prefer systematic enumeration, or a slightly more complicated probabilistic variant whose expected search time equals the one of systematic enumeration. Variants of systematic universal search were implemented in collaboration with Stefan Heil (1995) and Marco Wiering (1996). With the examples below, however, total search time is not the main issue: the simulations in the next section (based on probabilistic search) are intended to highlight generalization performance, not speed. Very similar results were obtained by systematic search, however. 4 APPLICATION: ....

[Article contains additional citation context not shown here]

Wiering, M. and Schmidhuber, J. (1996). Solving POMDPs with Levin search and EIRA. In Saitta, L., editor, Machine Learning: Proceedings of the Thirteenth International Conference, pages 534--542. Morgan Kaufmann Publishers, San Francisco, CA.

....is one of the simplest discrete methods. It will not solve nontrivial tasks #requiring many or precise parameters# in reasonable time. This explains our interest in more sophisticated discrete methods searching incrementally for better sequence processing algorithms such as Adaptive Levin Search #Wiering and Schmidhuber, 1996, 1997# based on Levin Search #Levin, 1973, 1984; Schmidhuber, 1997#, Genetic Programming with memory cells #e.g. Teller, 1994#, and Probabilistic Incremental Program Evolution #PIPE# with memory cells #Sa#lustowicz and Schmidhuber, 1997#. We will benchmark LSTM against PIPE and #nd that LSTM ....

Wiering, M. A. and Schmidhuber, J. #1996#. Solving POMDPs with Levin search and EIRA. In Saitta, L., editor, Machine Learning: Proceedings of the Thirteenth International Conference, pages 534#542. Morgan Kaufmann Publishers, San Francisco, CA.

....of Operations First consider a single learner. At time 0 (system birth) we initialize the learner s variable internal state I and its vector valued policy P ol. P ol s i th variable component is denoted P ol i . For now, there is no need to specify P ol i in some of our previous experiments [15, 12, 13, 14, 18, 20], however, P ol i typically is a variable probability distribution on a set of actions, and defines the conditional probability of executing a particular action, given current I. We introduce an initially empty stack S that allows for variable sized stack entries, and the conventional push and ....

....signals, and where no agent can speed up reinforcement intake by itself, this automatically enforces learning to cooperate . 4 RRL Experiments Due to space limitations, I will only be able to provide an overview of our numerous successful experiments with RRL. More details can be found in [15, 12, 13, 14, 18, 20]. Single agents. In recent work [18] we use Levin search (LS) 9] as a primitive action for an RRL based agent, which learns to solve partially observable Markov decision problems (POMDPs) POMDPs recently received a lot of attention in the reinforcement learning community. LS is theoretically ....

[Article contains additional citation context not shown here]

M. Wiering and J. Schmidhuber. Solving POMDPs with Levin search and EIRA. In L. Saitta, editor, Machine Learning: Proceedings of the Thirteenth International Conference. Morgan Kaufmann Publishers, San Francisco, CA, 1996. To appear.

....Methods from the second class search do not require EFs. Their policy space consists of complete algorithms defining agent behaviors, and they search policy space directly. Members of this class are Levin search (Levin, 1973; Levin, 1984; Solomonoff, 1986; Li and Vit anyi, 1993; Schmidhuber, 1995; Wiering and Schmidhuber, 1996), Genetic Programming (GP) Cramer, 1985; Dickmanns et al. 1987; Koza, 1992) and Probabilistic Incremental Program Evolution (PIPE) Sa lustowicz and Schmidhuber, 1997) Comparison. In our case study we compare two learning algorithms, each representative of its class: TD Q learning with linear ....

.... to help a bit since this focuses reinforcements on the best actions (although high greediness values do not work well either) Another yet untried option might be to use a pocket al..gorithm like method that stores good EFs and backtracks once performance decreases (e.g. the success story algorithm (Wiering and Schmidhuber, 1996; Schmidhuber and Zhao, 1997) 0 100 200 300 400 500 0 50 100 150 200 250 300 #games TD Q 11 players player (g=40) opponent (g=40) player (g=90) opponent (g=90) 0 0.2 0.4 0.6 0.8 1 0 50 100 150 200 250 300 #games TD Q 11 players TD Q (g=40) TD Q (g=90) Figure 8: Performance breakdown study: ....

[Article contains additional citation context not shown here]

Wiering, M. A. and Schmidhuber, J. (1996). Solving POMDPs with Levin search and EIRA. In Saitta, L., editor, Machine Learning: Proceedings of the Thirteenth International Conference, pages 534--542. Morgan Kaufmann Publishers, San Francisco, CA.

....the operation of the system. In this application, the environment of each connection s policy continually changes, because the policies of all the other connections keep changing. Note. This paper is based on [32] In the meantime we have published several additional papers on SSA, e.g. [50, 53, 40, 42]. 2 SSA for Incremental Self Improvement Outline. The evolutionary system in this section (see also [32] implements the ideas from section 1, in particular those in paragraph ( on incremental self improvement . To improve speed up its own (initially very dumb, highly random) learning ....

....To illustrate basic aspects of the principle, the remainder of this paper will also present a few experiments. These, how15 ever, in no way represent a systematic experimental analysis. In fact, many much more complex experiments are described in other recent papers on this subject, e.g. [50, 42, 40]. The experiments in the current section demonstrate that the system from section 2 indeed can learn to compute SSMs leading to faster and faster reinforcement intake. The system uses low level problem specific instructions in addition to the 17 general, assemblerlike instructions mentioned in ....

[Article contains additional citation context not shown here]

M.A. Wiering and J. Schmidhuber. Solving POMDPs with Levin search and EIRA. In L. Saitta, editor, Machine Learning: Proceedings of the Thirteenth International Conference, pages 534--542. Morgan Kaufmann Publishers, San Francisco, CA, 1996.

....which occupies 4. We set m = 50; n = 10. Given the primitives above, each animat faces a complex partially observable Markov decision problem (POMDP) e.g. Schmidhuber, 1991; Whitehead, 1992; Jaakkola et al. 1995; Kaelbling et al. 1995; Ring, 1994; McCallum, 1993; Littman et al. 1995; Wiering and Schmidhuber, 1996) the current input by itself does not necessarily provide all information needed to determine the optimal next action. Checkpoints. Like in section 3. for each animat, a checkpoint occurs after each 5th consecutive non zero reinforcement signal following each SMS (i.e N r = 5) Copying ....

....in this paper. Primitive actions can actually be almost anything. For instance, an action may correspond to a Bayesian analysis of previous events. While this analysis is running, time is running, too. Thus, the complexity of the Bayesian approach is automatically taken into account. Or, as in (Wiering and Schmidhuber, 1996), actions may be calls of Levin search (Levin, 1973) a theoretically optimal algorithm for a wide variety of non incremental search problems. Or, actions may be calls of a Q learning variant. This makes sense in situations where the applicability of Q learning is questionable because the ....

Wiering, M. and Schmidhuber, J. (1996). Solving POMDPs with Levin search and EIRA. In Saitta, L., editor, Machine Learning: Proceedings of the Thirteenth International Conference. Morgan Kaufmann Publishers, San Francisco, CA. To appear.

....MRL (MRL ALS for short) always outperforms ALS by itself. 4.4 Experiment 1: A Big, Partially Observable Maze (POM) The current section is a prelude to section 4.5 which will address inductive transfer issues. Here we will only show that LS by itself can be very useful for POMDP problems. See also (Wiering and Schmidhuber, 1996). Task. Figure 1 shows a 39 Theta 38 maze with a single start position (S) and a single goal position (G) The maze has many more fields and obstacles than mazes used by previous authors working on POMDPs for instance, McCallum s maze has only 23 free fields (McCallum, 1995) The goal is to ....

Wiering, M. and Schmidhuber, J. (1996). Solving POMDPs with Levin search and EIRA. In Saitta, L., editor, Machine Learning: Proceedings of the Thirteenth International Conference. Morgan Kaufmann Publishers, San Francisco, CA. To appear.

No context found.

Wiering, M. A. and Schmidhuber, J. (1996b). Solving POMDPs with Levin search and EIRA. In Saitta, L., editor, Machine Learning: Proceedings of the Thirteenth International Conference, pages 534--542. Morgan Kaufmann Publishers, San Francisco, CA.

....is highly unspecific in the sense that it can help to solve all kinds of typical , regular , interesting problems occurring in our atypical, regular universe. This paper, based on [1] goes beyond our other recent papers exploiting algorithmic regularities for practical machine learning purposes [9, 3, 10]. Levin search (LS) References [9, 3] show how a variant of Levin search (LS) 11, 12, 13, 8] can find problem solutions with low Kolmogorov complexity and high generalization ability in non random but otherwise quite general settings. LS is of interest because it has the optimal order of ....

....Then universal LS will solve the same problems in at most O(f(n) steps (although a large constant may be buried in the O( notation) Despite this strong result, until recently LS has not received much attention except in purely theoretical studies see, e.g. 14] Adaptive LS. References [15, 3, 10] note that LS is not necessarily optimal if algorithmic information (e.g. 8] between solutions to successive problems can be exploited to reduce future search costs based on experience. Our adaptive LS extension (ALS) 15, 10] does use experience to modify LS underlying probability ....

[Article contains additional citation context not shown here]

M.A. Wiering and J. Schmidhuber. Solving POMDPs with Levin search and EIRA. In L. Saitta, editor, Machine Learning: Proceedings of the Thirteenth International Conference, pages 534--542. Morgan Kaufmann Publishers, San Francisco, CA, 1996.

No context found.

M.A. Wiering and J. Schmidhuber. Solving POMDPs with Levin search and EIRA. In L. Saitta, editor, Machine Learning: Proceedings of the Thirteenth International Conference, pages 534--542. Morgan Kaufmann Publishers, San Francisco, CA, 1996.

....SSA cycle (SSA ALS for short) always outperforms ALS by itself. 3.4. Experiment 1: A Big Partially Observable Maze (POM) The current section is a prelude to section 3.5 which will address inductive transfer issues. Here we will only show that LS by itself can be useful for POE problems. See also (Wiering and Schmidhuber, 1996). Task. Figure 1 shows a 39 Theta 38 maze with 952 free fields, a single start position (S) and a single goal position (G) The maze has more fields and obstacles than mazes used by previous authors working on POMs for instance, McCallum s maze has only 23 free fields (McCallum, 1995) The ....

Wiering, M. and Schmidhuber, J. (1996). Solving POMDPs with Levin search and EIRA. In Saitta, L., editor, Machine Learning: Proceedings of the Thirteenth International Conference, pages 534--542. Morgan Kaufmann Publishers, San Francisco, CA.

....is one of the simplest discrete methods. It will not solve nontrivial tasks (requiring many or precise parameters) in reasonable time. This explains our interest in more sophisticated discrete methods searching incrementally for better sequence processing algorithms such as Adaptive Levin Search (Wiering and Schmidhuber, 1996, 1997) based on Levin Search (Levin, 1973, 1984; Schmidhuber, 1997) Genetic Programming with memory cells (e.g. Teller, 1994) and Probabilistic Incremental Program Evolution (PIPE) with memory cells (Sa lustowicz and Schmidhuber, 1997) We will benchmark LSTM against PIPE and find that LSTM ....

Wiering, M. A. and Schmidhuber, J. (1996). Solving POMDPs with Levin search and EIRA. In Saitta, L., editor, Machine Learning: Proceedings of the Thirteenth International Conference, pages 534--542. Morgan Kaufmann Publishers, San Francisco, CA.

Online articles have much greater impact   More about CiteSeer.IST   Add search form to your site   Submit documents   Feedback  

CiteSeer.IST - Copyright Penn State and NEC