Citation Context

...L and inductive transfer so exciting is that it provides hooks for so many different kinds of information that traditionally have been difficult to exploit. 9.2.11 Combining MTL and Boosting Bagging [=-=Breiman 1994-=-], Boosting [Schapire 1990][Freund 1995], Error-Correcting Codes [Dietterich & Bakiri 1995], and other voting schemes that combine multiple predictions for a task from different learned models are an ...

Citation Context

...ce comparable to MTL, but in some domains this works well.) Other approaches to missing data include marginalizing over CHAPTER 8. RELATED WORK 207 the missing values in learned probabilistic models [=-=Little & Rubin 1987-=-; Tresp, Ahmad & Neuneier 1994], and using methods like EM to iteratively reestimate missing values from current estimates of the data density [Ghahramani & Jordan 1994, 1997]. 8.8 Bayesian Graphical ...

Citation Context

...so exciting is that it provides hooks for so many different kinds of information that traditionally have been difficult to exploit. 9.2.11 Combining MTL and Boosting Bagging [Breiman 1994], Boosting [=-=Schapire 1990-=-][Freund 1995], Error-Correcting Codes [Dietterich & Bakiri 1995], and other voting schemes that combine multiple predictions for a task from different learned models are an exciting recent advance in...

Citation Context

...er ?? for discussion of how to optimize MTL nets for one task at a time.) An interesting, related approach to using multiple alternate output encodings for the same problem is error correcting codes [=-=Dietterich & Bakiri 1995-=-]. In this approach, the multiple encodings for the outputs are designed so that predictions from the multiple outputs can be combined such that the combined prediction is less sensitive to occasional...

Citation Context

...that it provides hooks for so many different kinds of information that traditionally have been difficult to exploit. 9.2.11 Combining MTL and Boosting Bagging [Breiman 1994], Boosting [Schapire 1990][=-=Freund 1995-=-], Error-Correcting Codes [Dietterich & Bakiri 1995], and other voting schemes that combine multiple predictions for a task from different learned models are an exciting recent advance in machine lear...

Citation Context

...the noise task is related to the other tasks. We may never have a theory of relatedness that allows us to reliably predict which tasks will help or hurt each other when used for inductive transfer. Because of this, we are now focussing part of our effort on ways of efficiently determining which tasks are beneficially related to each other. Of particular interest is recent work on feature selection that shows generalization performance sometimes improves if as many as half of the input features available on some of the large problems in the UCI repository are ignored, i.e., not used as inputs (Koller & Sahami, 1996; Liu & Setiono 1996). It would be interesting to test those problems to see if some of the “ignored” features might be well used as extra outputs (as was done in Section 4.9). 7.7. When Inductive Transfer Hurts MTL does not always improve performance. In the pneumonia domain, performance dropped for high-risk cases when an extra SSE output was added to the rankprop net (see Section 4.2). This was consistent with our model of the relative strengths and weaknesses of the main and extra task on this problem. MTL is a source of inductive bias. Some inductive biases help. Some inductive biases hur...

Citation Context

...king with the two medical domains. 2.1 1D-ALVINN 2.1.1 The Problem 1D-ALVINN uses a road image simulator developed by Pomerleau to permit rapid testing of learning methods for road-following domains [=-=Pomerleau 1992-=-]. The original simulator generates synthetic road images based on a number of user defined parameters such as road width, number of lanes, angle and field of view of the camera. We modified the simul...

Citation Context

... NETtalk is harder to learn using separate nets [Dietterich, Hild & Bakiri 1990; 1995]. A recent example of multiple tasks arising naturally is Mitchell's Calendar Apprentice System (CAP) [Dent 1992; =-=Mitchell et al. 1994-=-]. In CAP, the goal is to learn to predict the Location, Time Of Day, Day Of Week, and Duration of the meetings it schedules. These tasks are functions of the same data and share many common features....

Citation Context

...ues in learned probabilistic models [Little & Rubin 1987; Tresp, Ahmad & Neuneier 1994], and using methods like EM to iteratively reestimate missing values from current estimates of the data density [=-=Ghahramani & Jordan 1994-=-, 1997]. 8.8 Bayesian Graphical Models Models trained with supervised learning usually model only the likelihood of the output given the inputs, i.e., they usually learn only the conditional probabili...

Citation Context

...ve learner to prefer some hypotheses over other hypotheses. Bias-free learning is impossible; in fact, much of the power of an inductive learner follows directly from the power of its inductive bias [=-=Mitchell 1980-=-]. MTL is based on the notion that tasks can serve as mutual sources of inductive bias. CHAPTER 1. INTRODUCTION 32 0.76 0.78 0.8 0.82 0.84 0.86 0.88 0.9 0.92 0.94 0 1000 2000 3000 4000 5000 6000 7000 ...

Citation Context

...t are likely to be helpful. 1.2 MTL with Backprop Nets Hinton proposed that generalization in artificial neural networks improves if networks learn to represent underlying regularities of the domain [=-=Hinton 1986-=-]. A learner that learns many related tasks at the same time can use these tasks as inductive bias for each other and thus better learn the domain's regularities. This can make learning more accurate ...

Citation Context

...e computational cost of the experiments we run and to simplify the presen2 Other things such as K and the KERNEL W IDTH can also be shared for MTL. 3 This is the same assumption made in [O'Sullivan & =-=Thrun 1996-=-]. There, feature weights learned for a previous task are used for a new task when the number of samples for the new task is too small to support learning. This approach differs from MTL where the goa...

Citation Context

...endent problems trained in isolation. This is counterproductive--- related tasks can benefit each other if trained together. An early, almost accidental, use of multitask transfer in ANNs is NETtalk [=-=Sejnowski & Rosenberg 1986-=-]. NETtalk learns the phonemes and stresses to give a speech synthesizer to pronounce the words given it as inputs. NETtalk used one net with many outputs, partly because the goal was to control a syn...

Citation Context

...ed to the other tasks. We may never have a theory of relatedness that allows us to reliably predict which tasks will help or hurt each other when used for inductive transfer. Because of this, we are now focussing part of our effort on ways of efficiently determining which tasks are beneficially related to each other. Of particular interest is recent work on feature selection that shows generalization performance sometimes improves if as many as half of the input features available on some of the large problems in the UCI repository are ignored, i.e., not used as inputs (Koller & Sahami, 1996; Liu & Setiono 1996). It would be interesting to test those problems to see if some of the “ignored” features might be well used as extra outputs (as was done in Section 4.9). 7.7. When Inductive Transfer Hurts MTL does not always improve performance. In the pneumonia domain, performance dropped for high-risk cases when an extra SSE output was added to the rankprop net (see Section 4.2). This was consistent with our model of the relative strengths and weaknesses of the main and extra task on this problem. MTL is a source of inductive bias. Some inductive biases help. Some inductive biases hurt. It depends on the ...

Citation Context

...is NETtalk (Sejnowski & Rosenberg 1986). NETtalk learns the phonemes and stresses to give a speech synthesizer to pronounce the words given it as inputs. NETtalk used one net with many outputs, partly because the goal was to control a synthesizer that needed both phonemes and stresses at the same time. Although they never analyzed the contribution of multitask transfer to NETtalk, there is evidence that NETtalk is harder to learn using separate nets (Dietterich, Hild & Bakiri 1990, 1995). A more recent example of multiple tasks arising naturally is Mitchell’s Calendar Apprentice System (CAP) (Dent et al. 1992; Mitchell et al. 1994). In CAP, the goal is to learn to predict the Location, Time Of Day, Day Of Week, and Duration of the meetings it schedules. These tasks are functions of the same data and can share many common features. Early results using MTL decision trees (see Section 5.2) on this domain suggest that training these four tasks together yields better performance than training them in isolation, as is done in the CAP system. 4.8. Quantization Smoothing Often the world gives us quantized information. For example, the training signal may result from human assessment into one of several ca...

Citation Context

...ough training patterns for the new task are available to make optimizing attribute weights for the new task alone feasible. Combining MTL-KNN with TC should yield a method with the benefits of each. [=-=Breiman & Friedman 1995-=-] present a method called Curds & Whey that also combines sequential and parallel learning. Curds & Whey takes advantage of correlations between different prediction tasks. Models for different tasks ...

Citation Context

...e computational cost of the experiments we run and to simplify the presen2 Other things such as K and the KERNEL W IDTH can also be shared for MTL. 3 This is the same assumption made in [O'Sullivan & =-=Thrun 1996-=-]. There, feature weights learned for a previous task are used for a new task when the number of samples for the new task is too small to support learning. This approach differs from MTL where the goa...

Citation Context

... to reduce the computational cost of the experiments we run and to simplify the presen2 Other things such as K and the KERNEL W IDTH can also be shared for MTL. 3 This is the same assumption made in [=-=O'Sullivan & Thrun 1996-=-]. There, feature weights learned for a previous task are used for a new task when the number of samples for the new task is too small to support learning. This approach differs from MTL where the goa...

Citation Context

...r F , the extra information from T 0 about F will help the net learn F better from the other relationships. The simplest case of eavesdropping is when T = F . Abu-Mostafa calls these catalytic hints [=-=Abu-Mostafa 1990-=-]. In this case the net is being told explicitly to learn a feature F that is useful to the main task. Eavesdropping sometimes causes non-monotonic generalization curves for the tasks that eavesdrop o...

Citation Context

...t is learned by different tasks while tasks are trained in parallel is the central idea in multitask learning [Suddarth & Kergosien 1990; Dietterich, Hild & Bakiri 1990, 1995; Suddarth & Holden 1991; =-=Caruana 1993-=-a, 1993b, 1994, 1995; Baxter 1994, 1995, 1996; Caruana & de Sa 1996]. INPUTS . . . Task 1 Task 2 Task 3 Task 4 Figure 1.2: Multitask Backprop (MTL) of four tasks with the same inputs. Multitask learni...

Citation Context

...ile tasks are trained in parallel is the central idea in multitask learning [Suddarth & Kergosien 1990; Dietterich, Hild & Bakiri 1990, 1995; Suddarth & Holden 1991; Caruana 1993a, 1993b, 1994, 1995; =-=Baxter 1994-=-, 1995, 1996; Caruana & de Sa 1996]. INPUTS . . . Task 1 Task 2 Task 3 Task 4 Figure 1.2: Multitask Backprop (MTL) of four tasks with the same inputs. Multitask learning is a collection of learning al...

Citation Context

...ffect in a single learning task, but a very large effect in a continuing sequence of related learning tasks. Such cross-task learning may well be the key to powerful human-level learning abilities.&qu=-=ot; [Sutton 1992-=-] 8.3 Serial Transfer Transferring learned structure between related tasks is not new. The early work on sequential transfer of learned structure between neural nets [Pratt et al. 1991; Pratt 1992; Sh...

Citation Context

...rge sample would be needed to represent them with high fidelity), but the new sample of training data is small, it is beneficial to sample the prior model at more points than the current sample. See [=-=Craven & Shavlik 1994-=-] for a thorough discussion of synthetic sampling. It is interesting to note that it is relatively straightforward to use parallel transfer to do serial transfer, but it does not seem to be easy to us...

Citation Context

...m non-digits. [LeCun, 1997 (private communication)] 4.10 Sequential Transfer MTL is parallel transfer. It might seem that sequential transfer [Pratt & Mostow 1991; Pratt 1992; Sharkey & Sharkey 1992; =-=Thrun & Mitchell 1994-=-; Thrun 1995] should be easier. This may not be the case. Some of the advantages of parallel transfer are: ffl The full detail of what is being learned for all tasks is available to all tasks because ...

Citation Context

...y adding extra outputs to a backprop net might improve generalization performance. In Section 1.4 we mentioned several of these: ffl Adding noise to backpropagation sometimes improves generalization [=-=Holmstrom & Koistinen 1992-=-]. To the extent that tasks are uncorrelated, their contribution to the aggregate gradient (the gradient that sums the error fed back from each layer's outputs) can appear as noise to other tasks. Thu...

Citation Context

...learning abilities." [Sutton 1992] 8.3 Serial Transfer Transferring learned structure between related tasks is not new. The early work on sequential transfer of learned structure between neural n=-=ets [Pratt et al. 1991-=-; Pratt 1992; Sharkey & Sharkey 1992] clearly demonstrates that what is learned for one task can be used as a bias for other tasks. Unfortunately, this work failed to find improvements in generalizati...

Citation Context

...other tasks. Sharing what is learned by different tasks while tasks are trained in parallel is the central idea in multitask learning [Suddarth & Kergosien 1990; Dietterich, Hild & Bakiri 1990, 1995; =-=Suddarth & Holden 1991-=-; Caruana 1993a, 1993b, 1994, 1995; Baxter 1994, 1995, 1996; Caruana & de Sa 1996]. INPUTS . . . Task 1 Task 2 Task 3 Task 4 Figure 1.2: Multitask Backprop (MTL) of four tasks with the same inputs. Mu...

Citation Context

...ions that develop in the hidden layer for one task to be used by other tasks. Sharing what is learned by different tasks while tasks are trained in parallel is the central idea in multitask learning [=-=Suddarth & Kergosien 1990-=-; Dietterich, Hild & Bakiri 1990, 1995; Suddarth & Holden 1991; Caruana 1993a, 1993b, 1994, 1995; Baxter 1994, 1995, 1996; Caruana & de Sa 1996]. INPUTS . . . Task 1 Task 2 Task 3 Task 4 Figure 1.2: M...

Citation Context

...nterested in understanding what information is best used for supervised and unsupervised learning.) 8.6 Theories of Parallel Transfer Attempts have been made to develop theories of parallel transfer [=-=Abu-Mostafa 1993-=-; Baxter 1994, 1995, 1996]. The most advanced thus far is Baxter's theory for representation learning in neural nets. It shows that the number of training patterns required to learn N strongly related...

Citation Context

...997 (private communication)] 4.10 Sequential Transfer MTL is parallel transfer. It might seem that sequential transfer [Pratt & Mostow 1991; Pratt 1992; Sharkey & Sharkey 1992; Thrun & Mitchell 1994; =-=Thrun 1995-=-] should be easier. This may not be the case. Some of the advantages of parallel transfer are: ffl The full detail of what is being learned for all tasks is available to all tasks because all tasks ar...

Citation Context

...as speeding up learning. More recently, Mitchell and Thrun devised a serial transfer method called ExplanationBased Neural Nets (EBNN) [Thrun & Mitchell 1994; Thrun 1995, 1996] based on tangent prop [=-=Simard et al. 1992-=-] that yields improved generalization when trained on a sequence of learned tasks. EBNN trains the partial derivative of outputs with respect to the inputs at the same time the outputs are being train...

Citation Context

... tasks without committing to extra training information being available at run time, and also makes learning simpler in domains where some input features cannot reasonably be predicted. [Martin 1994, =-=Martin & Billman 1994-=-] explore how concept formation systems such as COBWEB can be extended to acquire overlapping concept descriptions. Their OLOC system is an incremental concept learner that learns overlapping probabil...

Citation Context

...entations. It should come as no surprise, then, that MTL is similar in some ways to other methods used for clustering and unsupervised learning. For example, small changes to the indices in COBWEB's [=-=Fisher 1987-=-] probabilistic information metric yields a metric suitable for judging splits in multitask decision trees. Whereas COBWEB considers all features as tasks to predict, MTL decision trees allow the user...

Citation Context

... learning to discriminate digits from non-digits. [LeCun, 1997 (private communication)] 4.10 Sequential Transfer MTL is parallel transfer. It might seem that sequential transfer [Pratt & Mostow 1991; =-=Pratt 1992-=-; Sharkey & Sharkey 1992; Thrun & Mitchell 1994; Thrun 1995] should be easier. This may not be the case. Some of the advantages of parallel transfer are: ffl The full detail of what is being learned f...

Citation Context

...ccomplishing what MTL accomplishes, or whether the extra complexity of their models will make them inferior to traditional supervised learning and MTL. 8.9 Other Uses of MTL 8.9.1 Committee Machines [=-=Munro & Parmanto 1997-=-] use extra tasks to improve the generalization performance of a committee machine that combines the predictions of multiple learned experts. Because committee machines work better if the errors made ...

Citation Context

...s on the outputs where each net has 30 outputs, but each net is optimized to perform best on one output at a time. CHAPTER 8. RELATED WORK 211 8.9.3 Task-Specific Selective Attention Baluja's thesis [=-=Baluja 1996-=-] considers the problem of training backprop nets on problems where knowing where to look improves recognition accuracy. For example, he trains backprop nets to locate and recognize +'s and x's in a s...

Citation Context

...to learn models to predict the lab CHAPTER 2. DOES IT WORK? 55 tests that will be missing, and provide these predictions as extra inputs to a net learning the main risk prediction task. Feature nets [=-=Davis & Stentz 1995-=-] is a competing approach to MTL that does this. It trains nets to predict the missing future measurements and uses the predictions, or the hidden layers learned for these predictions, as extra inputs...

Citation Context

...rincipled ways to determine what architecture is best for each problem. Fortunately, simple architectures often work CHAPTER 9. CONTRIBUTIONS, DISCUSSION, AND FUTURE WORK 217 well, if not optimally. [=-=Ghosn & Bengio 1997-=-] experiment with several different architectures for MTL in backprop nets. It might seem that deeper net architectures would work better with MTL-backprop; an extra hidden layer would make it possibl...

Citation Context

...te additional tasks without committing to extra training information being available at run time, and also makes learning simpler in domains where some input features cannot reasonably be predicted. [=-=Martin 1994-=-, Martin & Billman 1994] explore how concept formation systems such as COBWEB can be extended to acquire overlapping concept descriptions. Their OLOC system is an incremental concept learner that lear...

Citation Context

...ry assessment indicates further testing and hospitalization is warranted. But low-risk patients can often be identified using measurements made prior to hospitalization. The Medis Pneumonia Database [=-=Fine et al. 1993-=-] indicates whether each patient lived or died. 1,542 (10.9%) of the patients died. As before, the most useful decision aid for this problem would predict which patients will live or die. But this is ...

Citation Context

...ets with 200 hidden units and with 25 hidden units. This will tell us if generalization would be better with more or less capacity. Finally, we run a fourth experiment based on the heuristic used in [=-=Valdes-Perez & Simon 1994-=-] to discover complex patterns in data. In this experiment we shuffle the training signals (the target output values) for Tasks 2, 3, and 4 in the data set before training an MTL net on the four outpu...

Citation Context

... train these models, however, may no longer be available. Can MTL benefit from the prior learned models without the training data? Yes. One can use the model to generate synthetic data and use the training signals in the synthetic data as extra MTL tasks. This approach to sequential transfer elegantly sidesteps the catastrophic interference problem (forgetting old tasks while learning new ones), and is applicable even where the analytic methods of evaluating domain theories used by other serial transfer methods are MULTITASK LEARNING 59 not available. For example, EBNN (Thrun & Mitchell 1994; Thrun 1996) requires that the domain theory be differentiable, but the MTL approach to sequential transfer does not. This approach is most effective when the prior learned models are accurate. If the prior models are poor, they can be a poor source of inductive bias. Some serial transfer mechanisms have explicit mechanisms for reducing transfer when prior learning does not appear to be accurate for the task at hand (Thrun & Mitchell 1995; Thrun 1996). An issue that arises when synthesizing data from prior models is what distribution to sample from. One approach is to use the distribution of the training ...

Citation Context

...ional cost of learning nor the intelligibility of what is learned. One way transfer improves generalization is by providing a stronger inductive bias than would be available without the extra knowledge. This can yield better generalization with a fixed training set, or reduce the number of training patterns needed to achieve some fixed level of performance. Inductive bias is anything that causes an inductive learner to prefer some hypotheses over other hypotheses. Bias-free learning is impossible; much of the power of an inductive learner follows directly from the power of its inductive bias (Mitchell 1980). Multitask learning uses the training signals for related tasks as an inductive bias to improve generalization. One does not usually think of training signals as a bias; but when the training signals are for tasks other than the main task, it is easy to see that, from the point of view of the main task, the other tasks may serve as a bias. The multitask bias causes the inductive learner to prefer hypotheses that explain more than one task. For this multitask bias to exist, the inductive learner must be biased to prefer hypotheses that have utility across multiple tasks. 2. Does MTL Work? Befo...

Citation Context

...in Section 2 use a single fully connected hidden layer shared equally by all tasks. Sometimes, more complex net architectures work better. For example, sometimes it is beneficial to have a small private hidden layer for the main task, and a larger hidden layer shared by both the main task and extra tasks. But too many private hidden layers (e.g., a private hidden layer for each task) reduce sharing and the benefits of MTL. We do not currently have principled ways to determine what architecture is best for each problem. Fortunately, simple architectures often work well, even if not optimally. (Ghosn & Bengio 1997) experiment with several different architectures for MTL in backprop nets. Regularization methods such as weight decay can be used with MTL. By reducing the effective number of free parameters in the model, regularization promotes sharing. Too strong a bias for sharing, however, can hurt performance. If tasks are more different than they are alike (the usual case), it is important to allow tasks to learn fairly independent models and overlap only where there is common hidden structure. This is one reason why MTL performance often drops if the size of the shared hidden layer is much smaller tha...

Citation Context

...dependent problems trained in isolation. This is counterproductive| related tasks can bene t each other if trained together. An early, almost accidental, use of multitask transfer in ANNs is NETtalk [=-=Sejnowski & Rosenberg 1986-=-]. NETtalk learns the phonemes and stresses to give aspeechsynthesizer to pronounce the words given it as inputs. NETtalk used one net with many outputs, partly because the goal was to control a synth...

Citation Context

... discriminate digits from non-digits. [LeCun, 1997 (private communication)] 4.10 Sequential Transfer MTL is parallel transfer. It might seem that sequential transfer [Pratt & Mostow 1991; Pratt 1992; =-=Sharkey & Sharkey 1992-=-; Thrun & Mitchell 1994; Thrun 1995] should be easier. This may not be the case. Some of the advantages of parallel transfer are: ffl The full detail of what is being learned for all tasks is availabl...

Citation Context

... discriminate digits from non-digits. [LeCun, 1997 (private communication)] 4.10 Sequential Transfer MTL is parallel transfer. It might seem that sequential transfer [Pratt & Mostow 1991; Pratt 1992; =-=Sharkey & Sharkey 1992-=-; Thrun & Mitchell 1994; Thrun 1995] should be easier. This may not be the case. Some of the advantages of parallel transfer are: The full detail of what is being learned for all tasks is available to...

Citation Context

...s in Figure 1 were being trained on. Note that these four outputs are fully connected to a hidden layer that they share.1 Backpropagation is done in parallel on the four outputs in the MTL net. Because the four outputs share a common hidden layer, it is possible for internal representations that arise in the hidden layer for one task to be used by other tasks. Sharing what is learned by different tasks while tasks are trained in parallel is the central idea in multitask learning (Suddarth & Kergosien 1990; Dietterich, Hild & Bakiri 1990, 1995; Suddarth & Holden 1991; Caruana 1993, 1994, 1995; Baxter 1994, 1995, 1996; Caruana & de Sa 1996). MTL is an inductive transfer method that uses the domain specific information contained in the training signals of related tasks. It does this by learning the multiple tasks in parallel while using a shared representation. In backpropagation, MTL allows features developed in the hidden layer for one task to be used by other tasks. It also allows features to be developed to support several tasks that would not have been developed in any STL net trained on the tasks in isolation. Importantly, MTL also allows some hidden units to become specialized for just on...