R. Reed, Pruning algorithms - a survey, IEEE Transactions on Neural Networks, 4(5), 1993, 740--747.  Home/Search   Document Not in Database   Summary   Related Articles   Check

....but also methods that can model a decrease in the number of neurons, axons and synapses: Pruning is the process where links or units are removed from the network during training. These techniques are realised in various algorithms such as optimal brain surgeon, optimal brain damage and others [39]. Weight decay can be regarded as a selectionist method like pruning. However, instead of removing connections, it first restricts the growth of their weights by giving them a tendency to decay to zero, that is, the connection would disappear unless reinforced [20] Selective Learning with ....

....of all three forms of incremental learning might be employed. The following section reviews some experimental work of other researchers on incremental learning as well as results from our own experiments. 5. Example Studies While reviews of constructive studies are covered for example by [32, 34, 39] we focus on input or data incremental learning. The almost classical example study on input incremental learning was conducted by [8] where two versions of incremental learning were investigated which were entitled: incremental input and incremental memory. In the incremental input approach, ....

R. Reed. Pruning algorithms -- a survey. IEEE Transactions on Neural Networks, (4):740--747, 1993.

....One involves the use of larger network architecture at the beginning and pruning it down to near optimum size. Learning algorithms using this general approach are called pruning. Examples include optimal brain damage[1] optimal brain surgeon[2] interactive pruning[3] and skeletonization[4] See[5] and[6, chapter 13] for a good review of pruning algorithms. With the other approach, the training begins with a minimal network and ends with a satisfactory network size. The algorithms using this approach are referred to as growth or constructive methods. Examples include the cascade correlation ....

Reed, R. Pruning algorithms - a survey," IEEE transactions on Neural Networks, 4(5):740-747, September 1993.

.... hidden units and add new units, or split existing units if the performance of the network is not satisfactory [Lee 1991, Wynne Jones 1992] Conversely, the main approach of pruning algorithms is to train a network that is larger than necessary and then to remove redundant elements [Hagiwara 1993, Reed 1993, Weigend 1993] In this paper we consider network pruning, specifically pruning of feedforward neural network architectures. Pruning algorithms may elect to remove weights and or units. We present a sensitivity analysis pruning algorithm to prune the input and hidden layers of a trained ....

R Reed, Pruning Algorithms -- A Survey, IEEE Transactions on Neural Networks, Vol. 4, No. 5, pp 740--747, 1993.

....to enforce smoothness or some other desirable characteristic of the function to be realized. Similar e ects can be achieved through early stopping of the training process. The second approach is to either add or delete hidden units as training proceeds, leading to growing or pruning algorithms [GT94, Ree93]. These two approaches are examined in the next two sections. 5.1 Regularization of RBFNs Regularization is a powerful technique to favor certain solutions over others by adding a penalty functional to the original cost function. The penalty embodies Occam s Razor in some form by penalizing more ....

R. Reed. Pruning algorithms{a survey. IEEE Transactions on Neural Networks, 4(5):740-747, September 1993.

....The minimal number of parameters which makes the model capable of approximating the underlying process can be considered as the optimal number of parameters. A neural network model with the optimal number of parameters can be obtained by the network growing [2 4] or the network pruning techniques [5]. While the network growing techniques start with a small network and gradually increase its size, adding one by one new neurons during the course of training, the network pruning techniques prune the redundant and insignificant parameters and hidden neurons from a large network. Obviously, ....

Reed, R.: Pruning Algorithms-A Survey. IEEE transactions on Neural networks, vol. 4, no. 5 (1993), p. 740- 747.

....it is best to moderate it with a component of the previous DeltaE value during pruning: DeltaE t 1 j = 0:8 DeltaE t j 0:2 E t ff j (46) Further details and examples are given in Mozer and Smolensky [98] 5. 4 Summary These and other pruning techniques are surveyed further in Reed [113]. To complete the pruning process we also need to have an effective stopping criterion. So far we have suggested minimisation of the training error until less than a prescribed tolerance. Though a common stopping criterion, the training error can introduce a bias due to the choice of dataset. ....

R. Reed. Pruning algorithms - a survey. IEEE Transactions on Neural Networks, pages 740--747, 1993.

....1. There are several explicit and implicit possibilities for reducing the number of parameters in the network. Penalty terms like weight decay belong to the class of implicit method which we have not considered here. An explicit approach to reduce the number of parameters is pruning (see e.g. [5], 6] Weight pruning removes individual connections whereas node pruning removes complete input or hidden nodes with all their associated parameters. We have used a method called epsi pruning [6] however other pruning methods will not change the results qualitatively. Empirically, based on ....

Russell Reed. Pruning algorithms - a survey. IEEE Transactions on neural networks, 4(5):740-- 747, 1993.

....of neural networks (NNs) is still a largely unsolved problem. Constructive and destructive algorithms attempt to offer a solution to the problem, by beginning with a small network and adding new features as needed, or by starting from a large network and removing unnecessary elements, respectively [1, 2, 3]. However, both approaches constrain the architectures achieved, either from the beginning, or through the structural modifications they introduce. Recently, new promising approaches based on evolutionary algorithms, such as evolutionary programming (EP) 4] and genetic algorithms (GAs) 5] have ....

R. Reed. Pruning algorithms: a survey. IEEE Transactions on Neural Networks, 4(5):740--747, 1993.

....techniques and can be easily adapted to a large variety of architectures and algorithms. Secondly, they do not require any a priori assumptions about the model (network) or training data all pruning algorithms require prior assumptions, some of which are difficult to justify see for example [28]. Thirdly, they are easy to understand and implement. Finally, and most importantly for this paper, they work very well with ensembles. For example, as we will show in this paper they allow diversity to be easily tuned, which as discussed in section 3, is a very important consideration for bagged ....

R. Reed, Pruning algorithms -- a survey, IEEE Transactions on Neural Networks 4 (1993) 740-747.

.... policies dominate all other variable horizon policies (including KNN) A problem with these policies is the determination of a good network topology, although more sophisticated learning algorithms exist which add weights and or units or remove weights and or units from the network (see [Reed, 1993]) An other disadvantage is the rather slow learning rate of the multi layered perceptrons. In the two layered perceptrons that we used, the number of hidden units determines the number of decision boundaries the neural net is able to learn. Training of a neural net results in fact in positioning ....

REED, R [1993], Pruning algorithms - A survey, IEEE Transactions on Neural Networks 4, 740--747.

.... corresponding to w f Gammaig , i.e. training a new network with one input less, is estimated using the weights w f Gammaig by Pw f Gammaig (X f Gammaig ) P w f Gammaig (X f Gammaig ) Approximate retraining algorithms have a flavor of complexity reduction algorithms (see e.g. [17, 18]) Each reduction step, which removes the least relevant weight or set of weights, requires an estimate of the change in performance due to this reduction. Approximate retraining can be viewed as a one step complexity reduction algorithm where the set of weights to be removed consists of all ....

....determination can be based on a training set and does not have to waste valuable data for a test set. However, a test set can be very useful to determine when to stop removing variables. Several suggestions in this direction can be found in the literature, both on pruning algorithms (see e.g. [17]) and on subset selection (see e.g. 4, 31] The relevance of information is also influenced by the effort needed to extract this information [34] Effort is a negative factor: other things being equal, the greater the effort, the lower the relevance. In this article, we have assumed that the ....

Russell Reed. Pruning algorithms - a survey. IEEE Transactions on Neural Networks, 4(5):740--747, 1993.

.... policies dominate all other variable horizon policies (including KNN) A problem with these policies is the determination of a good network topology, although more sophisticated learning algorithms exist which add weights and or units or remove weights and or units from the network (see Reed [1993]) An other disadvantage is the rather slow learning rate of the multi layered perceptrons. In the two layered perceptrons that we used, the number of hidden units determines the number of decision boundaries the neural net is able to learn. Training of a neural net results in fact in positioning ....

REED, R [1993], Pruning algorithms - A survey, IEEE Transactions on Neural Networks 4, 740--747.

....Bochum, FRG 5 task s complexity in terms of hyperplanes and the used nonlinearities can be drawn from the figures. Excluding the neurons with a comparable low gain or an uncharacteristic gain evolution corresponds to pruning the network. However, in contrast to established pruning techniques [6], this approach does not depend on additional assumption such as low absolute weight values or the frequency of effective updates of the weights. The necessary information is drawn from the learning dynamics itself. A visualization like ours permits a direct comparison of the learning speed of ....

R. Reed. Pruning Algorithms -- A Survey. IEEE Transactions on Neural Networks, Vol. 4, No. 5:pp.740--747, 1993.

....disrupt the learning mechanism of the classifier. The selection of the network size for example is of paramount importance [18] Since the optimal size is not known in advance, there are a multitude of methods that grow or prune networks (a process that removes units or weights) during training [50, 61, 113, 128]. An alternative to modifying the size of the network is to force certain weight values to decay, effectively removing them [27, 88, 96] Another factor that affects the end result, is the training regime. Although mean square error minimization is the most common approach, there are other ....

R. Reed, Pruning algorithms---A survey, IEEE Transactions on Neural Networks, 4 (1993), pp. 740--747.

.... Techniques for reducing the number of parameters are greedy constructive learning [7] pruning [5, 12, 14] or weight sharing [18] Techniques for reducing the size of each parameter dimension are regularization, such as weight decay [13] and others [25] or early stopping [17] See also [8, 20] for an overview and [9] for an experimental comparison. Early stopping is widely used because it is simple to understand and implement and has been reported to be superior to regularization methods in many cases, e.g. in [9] 1.2 The basic early stopping technique In most introductory papers ....

Russel Reed. Pruning algorithms --- a survey. IEEE Transactions on Neural Networks, 4(5):740--746, 1993.

....of Experts Test set MSE 1 0.0100 2 0.0090 3 0.0087 4 0.0083 Table 3: Growing Mixture of Experts on the Multivariate Data set. Number of Experts Ave. MSE Static 15 0.0107 Architecture 20 0.0093 25 0.0085 30 0.0089 Network Growth 15.5 0. 0077 links have been proposed, especially for the MLP network [20]. Here, a simple and efficient technique to structurally adapt (prune and grow) the mixture of experts network is presented wherein the localized nature of the chosen gating network is once again exploited. From section 2.3, it is observed that the prior ff j is computed in the M step as ff ....

....complex regions of expertise. However, when an MLP is employed as the gating or expert network, the advantage of being able to train the network in one pass in the M step of the EM iterations is lost. Model selection via network growing or pruning has been widely studied for feedforward networks [20], 19] 17] In the mixture of experts framework, Waterhouse et al. 18] have developed a growing technique to build a hierarchical mixture of experts, based on principles from Classification and Regression Trees [27] The hierarchical network is grown one layer at a time by splitting an expert at ....

R. Reed. Pruning algorithms - a survey. IEEE Transactions on Neural Networks, 4(5):740-- 747, 1993.

.... of (almost) optimal neural networks have been proposed (see [7] 13] and [1] for overviews) Some of those approaches are aimed at improving well known architectures by training a network which is expected to be big enough to solve the given problem and subsequently remove (prune) units [15] [11]. The main problem with this approach is that big enough is an a priori indeterminable quantity and therefore the training of a network involves a lot of trial and errors. The number of trials is often further increased, it is unclear whether the network does not learn, due to either a bad choice ....

Russell Reed. Pruning Algorithms -- A Survey. IEEE Transactions on Neural Networks, vol. 4, num. 5, pp. 740--747, 1993.

....keeping their fitness intact and maintaining their principal functional characteristics. 1 Introduction Pruning a neural network is a standard approach in the connectionist literature by which unimportant weights are removed, usually in order to enhance the network s generalization performance [1]. In this paper we focus on pruning and its applications in the context of Evolved Autonomous Agents (EAA) To demonstrate this we propose a new network minimization algorithm, Evolutionary Network Minimization (ENM) ENM prunes successful networks while maintaining their fitness and principal ....

Reed, R.: Pruning algorithms - a survey. IEEE Transactions on Neural Networks 4 (1993) 740--747

....keeping their fitness intact and maintaining their principal functional characteristics. I Introduction Pruning a neural network is a standard approach in the connectionist literature by which unimportant weights are removed, usually in order to enhance the network s generalization performance [1]. In this paper we focus on pruning and its applications in the context of Evolved Autonomous Agents To demonstrate this we propose a new network minimization algorithm, Evolu tionary Network Minimization (ENM) ENM prunes successful networks while maintaining their fitness and principal ....

Reed, R.: Pruning algorithms - a survey. IEEE Transactions on Neural Networks 4 (1993) 740-747

No context found.

R. Reed, Pruning algorithms - a survey, IEEE Transactions on Neural Networks, 4(5), 1993, 740--747.

No context found.

R. Reed. Pruning algorithms - a survey. IEEE Transactions on Neural Networks, 4(5):740-- 747, 1993.

No context found.

Reed, R.: Pruning algorithms---a survey. IEEE Transactions on Neural Networks 4 (1993) 740--747

No context found.

Russel Reed. Pruning algorithms --- a survey. IEEE Transactions on Neural Networks, 4#5#:740#746, 1993.

No context found.

R. Reed. Pruning algorithms - a survey. IEEE Transactions on Neural Networks, 4(5):740-747, 1993.

No context found.

R. Reed. Pruning algorithms: a survey. IEEE Transactions on Neural Networks, 4(5):740-- 747, 1993.

First 50 documents

Online articles have much greater impact   More about CiteSeer.IST   Add search form to your site   Submit documents   Feedback  

CiteSeer.IST - Copyright Penn State and NEC