Caruana, Rich. 1995. Learning many related tasks at the same time with backpropagation. In Advances in Neural Information Processing Systems, volume 8, pages 657--664.  Home/Search   Document Details and Download   Summary   Related Articles   Check

.... Recently, this need has been recognised in the machine learning community by initiating a new research direction, termed learning to learn [31] Transfer in supervised learning involves reusing the features developed for one classi cation or prediction task as a bias for learning related tasks [3, 4, 30]. Transfer in reinforcement learning involves reusing the information gained while learning to achieve one goal for learning to achieve other goals more easily [25, 30] At the time of writing, only very few of these approaches have been applied to real robots [30] Ring [25] de ned continual ....

R. A. Caruana. Learning many related tasks at the same time with backpropagation. In G. Tesauro, D. Touretzky, and T. Leen, editors, Advances in Neural Information Processing Systems, volume 7, pages 657-664. MIT Press, 1995.

....in complexity [7, 30, 10] A number of approaches have been proposed for scaling up learning, by introducing constraints in addition to observed training data. These include, for example, engineering human knowledge into the system [29, 20] and using constraints from related learning tasks [21, 1]. This chapter considers methods by which a robot can use previously learned knowledge about its environment to augment the available training data when confronting new perceptual learning tasks. Figure i represents this framework. Inductive Learner Door Recognizer Knowledge EBNN Learner ....

Rich Caruana. Learning Many Related Tasks At the Same Time With Backpropagation. In Advances in Neural Lnformation Processing Systems 6. Morgan Kaufmann, December 1994.

....When AutoDJ generates playlists, the user may select only one training example. No useful similarity metric can be derived from one training example, so AutoDJ uses meta training to learn the kernel. The idea of meta training comes from the learning to learn or multi task learning literature [2, 5, 10, 13]. This paper is most similar to Minka Picard [10] who also suggested fitting a mean and covariance for a Gaussian Process based on related functions. However, in [10] in order to generalize the covariance beyond the meta training points, a Multi Layer Perceptron (MLP) is used to learn multiple ....

R. Caruana. Learning many related tasks at the same time with backpropagation. In NIPS, volume 7, pages 657--664, 1995.

....The theory of learning biases to guide subsequent learning is a focus of current research in the machine learning community. Some basic mechanisms which have been found to allow separate tasks to contribute biases are: ffl Features created for predicting one task can be reused for novel tasks [15, 13, 2, 5]. ffl Features shared across multiple tasks are more robust to noise in the training data[5] ffl Features shared across multiple tasks are more likely to capture true regularities of a domain[2, 5] ffl Features useful to multiple tasks can be easier to learn in one task T than another T 0 , ....

....learning community. Some basic mechanisms which have been found to allow separate tasks to contribute biases are: ffl Features created for predicting one task can be reused for novel tasks [15, 13, 2, 5] ffl Features shared across multiple tasks are more robust to noise in the training data[5]. ffl Features shared across multiple tasks are more likely to capture true regularities of a domain[2, 5] ffl Features useful to multiple tasks can be easier to learn in one task T than another T 0 , due to T 0 using features in a more complex way[1, 15] In this paper, we build on these ....

[Article contains additional citation context not shown here]

R. Caruana. Learning Many Related Tasks At the Same Time With Backpropagation. In Advances in Neural Information Processing Systems 6. Morgan Kaufmann, December 1994.

....2.2 Internal State Sharing Instead of re representing knowledge, some knowledge reuse research has focused on reusing internal state information, namely weight values in MLP style neural networks. Under the belief that related classi cation tasks may bene t from common internal features, Caruana [6] has created an MLP based multiple classi er system that is trained simultaneously to perform several related classi cation tasks. In this work on two layer neural networks, the rst layer is shared by several related classi cation tasks. The premise is that related tasks have similar ....

Caruana, R. (1995). Learning many related tasks at the same time with backpropagation. In Advances in Neural Information Processing Systems 7, pages 657-664.

....of x space. The art is to choose the common and separate parameters and the parameter priors (i.e. the regularization) guided by available prior knowledge or assumptions. Response coaching can be seen as a special case of the problem of learning how to learn or multitask learning (e.g. 73] 4] [15], 85] 61] The relationship x 7 z is a second task which is learned together with the primary one in an attempt to employ information ow through latent, shared variables. A very general approach to this problem is suggested in [66] the author refers to the problem as family discovery ) ....

Rich Caruana. Learning many related tasks at the same time with backpropagation. In Advances in Neural Information Processing Systems 7. MIT Press, 1995.

....possible to use a similar technique in other weight based classifier architectures. Multi task Learning Building on the concept of learning from hints in [1] Caruana has studied an MLP type of neural network architecture where several related classification problems share a common first layer [16]. The idea, known as multi task learn 17 ing (MTL) is that there are shared relevant concepts which can be represented by the weights values of a common layer. Training occurs simultaneously for all tasks, and during the training process, regions in the shared layer defining these common ....

Rich Caruana. Learning many related tasks at the same time with backpropagation. In Advances in Neural Information Processing Systems 7, pages 657--664, 1995.

....(or at worst equal) solution than the independent regressors. Appropriate weights may be estimated from data [3] There do not exist to our knowledge similar results in the case of non linear function approximation. However, a number of empirical investigations exist in the case of classification [2]. To test the interest of integrated function regressors, we built a single NN to estimate simultaneously the thirty five functions needed for the computation of the intensity of radiation at a given frequency. Instead of using 35 MLPs with one output, we utilized a (fully connected, one hidden ....

....of selectively improving the accuracy of the functions that play an important role in the computation of the global solution. References [1] Rivire Ph. Soufia A. and Taine J. 1992) Correlated k and fictitious gas methods for H2O near 2.7m, J. Quant. Spetrosc. Radiat. Transfer 48(2) 187 203. [2] Caruana R. 1994) Learning Many Related Tasks at the Same Time With Backpropagation, Advances in Neural Information Systems 7, 664 657. 3] Breiman L. and Friedman, J.H. Predicting Multivariate Responses in Multiple Linear Regression. Royal Statistical Society (in press) 4] Zaragoza H. ....

[Article contains additional citation context not shown here]

Caruana R. (1994). Learning Many Related Tasks at the Same Time With Backpropagation, Advances in Neural Information Systems 7, 664-657.

....in complexity [7, 30, 10] A number of approaches have been proposed for scaling up learning, by introducing constraints in addition to observed training data. These include, for example, engineering human knowledge into the system [29, 20] and using constraints from related learning tasks [21, 1]. This chapter considers methods by which a robot can use previously learned knowledge about its environment to augment the available training data when confronting new perceptual learning tasks. Figure 1 represents this framework. Examples Inductive Learner Door Recognizer Domain ....

Rich Caruana. Learning Many Related Tasks At the Same Time With Backpropagation. In Advances in Neural Information Processing Systems 6. Morgan Kaufmann, December 1994.

....reflects the property of a shared domain s regularity, if a correct additional hint is given. An improvement of generalization performance has been observed in other domain s applications [Caruana, Baluja, Mitchell 1996] Different underlying mechanisms have been discussed in [Abu Mostafa 1995] [Caruana 1995]. 3 Application An interesting field for Multitask Learning and the use of hints is the domain of financial forecasting. Here an application of predicting the direction of five German stocks in a one week horizon was chosen: Allianz, Daimler Benz, Deutsche Bank, Siemens and Veba. Because of its ....

Caruana, R.: Learning many related tasks at the same time with backpropagation, Advances in Neural Information Processing Systems, 7,656-664, 1995

....extrahieren und in geeigneter Form in das Lernen dieser anderen Zielvariablen zu transferieren. Das Kernproblem des Mehraufgaben TransferLernens ist dabei die Frage, wie der Ergebnistransfer von Aufgabe zu Aufgabe durchzuf uhren ist. Diese Frage ist bisher nur f ur neuronale Netze beantwortet, wo (Caruana, 1994) nachgewiesen hat, da ein Transfer zur Verbesserung der Klassifikationsgenauigkeit f uhren kann. F ur eine der wichtigsten und popul arsten Klassen von Lernverfahren, den Entscheidungsbaumverfahren, existiert ein solcher Nachweis bisher nicht. Dieser Artikel berichtet daher detailliert uber ....

Caruana, Richard A. (1994). Learning many related tasks at the same time with backpropagation. In: Advances in neural Information Processing Systems 7 (Proceedings of NIPS*94). pp. 657--664.

....network for each action. RLds and RLps refer to the distributed and parallel RL architectures with separate action networks. Tables 6.12, 6.13, and 6.14 show a comparison of the performance of the combined versus separate action networks, trained under the same conditions. In spite of experiments [25] that seem to suggest that combined networks should be superior, in this particular problem that does not appear to be the case. Table 6.12 Combined versus separate action network results for down peak profile with down traffic only. Algorithm AvgWait SquaredWait SystemTime Percent 60 secs RLp ....

Caruana, R. Learning many related tasks at the same time with backpropagation. In Tesauro, G., Touretzky, D., and Leen, T., editors, Advances in Neural Information Processing Systems 7. MIT Press, Cambridge, MA, 1995.

.... Functional transfer does not involve the explicit assignment of prior task representation to a new task, rather it employs the use of implicit pressures from supplemental training examples [1, 33] the parallel learning of related tasks constrained to use a common internal representation [6, 9], or the use of historical training information (most commonly the learning rate or gradient of the error surface) to augment the standard weight update equations [20, 23, 34] These pressures serve to reduce the effective hypothesis space in which the learning system performs its search. This ....

....value from the perspective of increased generalization performance. Certain methods of functional transfer have also been found to reduce training time (measured in number of training iterations) Chief among these methods is the parallel MTL paradigm explored recently by Caruana and Baxter [6, 9]. Critical to the transfer of knowledge from pool of source tasks to a primary task is some measure of relatedness between those tasks. Relatedness is a difficult subject which gets one into the philosophy of similarity, analogy, and metaphor. Nonetheless, most individuals working on transfer in ....

[Article contains additional citation context not shown here]

Richard A. Caruana. Learning many related tasks at the same time with backpropagation. Advances in Neural Information Processing Systems 7, 7:657--664, 1995.

.... in the past that training a classifier on multiple tasks (using the same data) may be an efficient way to introduce desirable bias into the solution (Caruana, 1993) Our motivation for multiple task training is, however, fundamentally different from the subsequent development of that idea by Caruana (1995), who implicitly assumes that the different tasks are on the same level of categorization. In comparison, our approach calls for internal representation to be constructed using a combination of various tasks, including tasks at different levels of categorization. For example, if required to create ....

Caruana, R. (1995). Learning many related tasks at the same time with backpropagation. In Tesauro, G., Touretzky, D., and Leen, T., editors, Advances in Neural Information Processing Systems, volume 7, pages 657--664. Morgan Kaufmann, San Mateo, CA.

....Some recent work in knowledge reuse has focused on the automated extraction and reuse of knowledge from the data sets of other relevant classifiers, including reuse of the trained classifiers themselves. Under the belief that related classification tasks may benefit from common internal features, Caruana[Caruana, 1995] has created a multilayer perceptron (MLP) based multiple classifier system that is trained simultaneously to perform several related classification tasks. The first layer of the MLP is common to all tasks and the second layer is specific to individual tasks. The first layer is expected to learn ....

Caruana, R. (1995). Learning many related tasks at the same time with backpropagation. In Advances in Neural Information Processing Systems 7, pages 657--664.

....Bias Inductive Induced Model of Classifier Task Environment Figure 1: The framework for knowledge based inductive learning. use of implicit pressures from supplemental training examples [AM95, Sudd90] the parallel learning of related tasks constrained to use a common internal representation [Baxt95, Caru95], or the use of historical training information (most commonly the learning rate or gradient of the error surface) to augment the standard weight update equations [Mitc93, Naik93, Thru94, Thru95a] These pressures serve to reduce the effective hypothesis space in which the learning system performs ....

....value from the perspective of increased generalization performance. Certain methods of functional transfer have also been found to reduce training time (measured in number of training iterations) Chief among these methods is the parallel MTL paradigm explored recently by Caruana and Baxter [Baxt95, Caru95]. A recent paper by Caruana [Caru97] expresses plans for research into the use of MTL networks for sequential learning. We encourage these efforts as this is a large and exciting area of scientific discovery. 2.3 MTL Network Learning Kehoe points out in [Keho88] that psychological studies of ....

[Article contains additional citation context not shown here]

Richard A. Caruana, "Learning many related tasks at the same time with backpropagation ", Advances in Neural Information Processing Systems 7, Morgan Kaufmann, Vol. 7, pp. 657--664, San Mateo, CA, 1995.

....(or at worst equal) solution than the independent regressors. Appropriate weights may be estimated from data [3] There do not exist to our knowledge similar results in the case of non linear function approximation. However, a number of empirical investigations exist in the case of classification [2]. To test the interest of integrated function regressors, we built a single NN to estimate simultaneously the thirty five functions needed for the computation of the intensity of radiation at a given frequency. Instead of using 35 MLPs with one output, we utilized a (fully connected, one hidden ....

....selectively improving the accuracy of the functions that play an important role in the computation of the global solution. References [1] Rivi re Ph. Soufia A. and Taine J. 1992) Correlated k and fictitious gas methods for H2O near 2.7 m, J. Quant. Spetrosc. Radiat. Transfer 48(2) 187 203. [2] Caruana R. 1994) Learning Many Related Tasks at the Same Time With Backpropagation, Advances in Neural Information Systems 7, 664 657. 3] Breiman L. and Friedman, J.H. Predicting Multivariate Responses in Multiple Linear Regression. Royal Statistical Society (in press) 4] Zaragoza H. ....

[Article contains additional citation context not shown here]

Caruana R. (1994). Learning Many Related Tasks at the Same Time With Backpropagation, Advances in Neural Information Systems 7, 664-657.

No context found.

Caruana, Rich. 1995. Learning many related tasks at the same time with backpropagation. In Advances in Neural Information Processing Systems, volume 8, pages 657--664.

No context found.

R. Caruana. Learning many related tasks at the same time with backpropagation. In Adv. in Neural Info. Proc. Systems, volume 7, pages 657--674, 1995.

No context found.

Caruana, R. (1995). Learning many related tasks at the same time with backpropagation. Advances in Neural Information Processing Systems 7 (Proceedings of NIPS-94) (pp. 657-664).

No context found.

R. Caruana. Learning many related tasks at the same time with backpropagation. In G. Tesauro, D. Touretzky, and T. Leen, editors, Advances in Neural Information Processing Systems 7, pages 657-664. The MIT Press, 1995.

No context found.

R. Caruana. Learning many related tasks at the same time with backpropagation. In Adv. in Neural Info. Proc. Systems, volume 7, pages 657--674, 1995.

No context found.

R. A. Caruana. Learning many related tasks at the same time with backpropagation. In G. Tesauro, D. Touretzky, and T. Leen, editors, Advances in Neural Information Processing Systems, volume 7, pages 657--664. MIT Press, 1995.

No context found.

Caruana R.: Learning Many Related Tasks at the Same Time with Backpropagation, in: Advances in Neural Information Processing Systems (NIPS 1994), Tesauro G., Touretzky D. and Leen T.

No context found.

Richard Caruana. Learning Many Related Tasks at the Same Time with Backpropagation. In Advances in Neural Information Processing 5, 1993.

First 50 documents

Online articles have much greater impact   More about CiteSeer.IST   Add search form to your site   Submit documents   Feedback  

CiteSeer.IST - Copyright Penn State and NEC