T.-Y. Kwok and D.-Y Yeung, "Objective Functions for Training New Hidden Units in Constructive Neural Networks," IEEE Trans. on Neural Networks, vol. 8, no. 5, Sept. 1997.  Home/Search   Document Details and Download   Summary   Related Articles   Check

....of h t 1 and Q t the parmeters set of h t . The updating process usually involves two main operations, modify the existing parameters of Q t 1 and add new parameters if needed and learn the new set Q t . Incremental learning should be differentiated from online learning and constructive learning [21] which convey other meanings in the machine learning community: in opposition to batch learning algorithms, online learning algorithms try to learn and update the current classifier using only the new available data, that is, without using any past observed data; constructive learning aims at ....

....designing a neural network of appropiate size, given the full training dataset. However, in both of these domains, automatic growing of the system (network) and adding new units have been studied. Especially, the strategies referred by Kwok et al. in their survey about constructive algorithms ([21]) consists in re learning the whole parameters set or only the added unit when adding a new unit. A natural extension to the incremental learning of a static concept is to consider that the target concept or function changes with times. In this case, p(x;y) is time dependent and can be written p ....

T. -Y. Kwok and D. -Y. Yeung. Objective Functions for Training New Hidden Units in Constructive Neural Networks. IEEE Transactions on Neural Networks, vol. 8, number 5, pp. 1131--1148, http://citeseer.nj.nec.com/kwok99objective.html, 1997.

....we can now proceed to retrain the neural network. One of the important issues related to novelty detection is how to retrain the neural network. Incremental rather than complete retraining is important to decrease the overall computational load in this context. Constructive neural networks [12] and other approaches have appeared in literature to incrementally increase the number of classes and hidden nodes without completely new retraining. In our approach, we use a new neural network with a larger number of output nodes but the same architecture otherwise. The weights on links that ....

T. Kwok, D. Yeung, "Objective functions for training new hidden units in constructive neural networks", IEEE Transactions on neural networks, vol. 8 no. 5, pp. 1131-1148.1999.

.... Task: It is desirable that a learning algorithm construct networks whose complexity (in terms of relevant criteria such as number of nodes, number of links, and connectivity) is commensurate with the intrinsic complexity of the underlying learning task (implicitly specified by the training data) [26]. Constructive algorithms search for small solutions first and thus offer a potential for discovering a nearminimal network that suitably matches the complexity of the learning task. Smaller networks are also preferred because of their potential for more efficient hardware implementation and ....

T.-Y. Kwok and D.-Y. Yeung, "Objective functions for training new hidden units in constructive neural networks," IEEE Trans. Neural Networks, vol. 8, pp. 1131--1148, 1997.

....on standard benchmark data (Section 7.1) The second set of experiments measure the complexity and accuracy of the rules extracted by REFANN (Section 7.2) 7. 1 N2PFA Results There have been a number of papers that propose algorithms for constructing and or training neural network for regression [6, 30, 31, 32, 33]. A recent paper by Frank et al. 34] compares the results of naive Bayes methodology for regression to those from other regression methods including linear regression. Test results from 32 problems are reported in the paper. The data sets are available from their website ....

T. Y. Kwok, and D. .Y. Yeung, \Objective functions for training new hidden units in constructive neural networks," IEEE Trans. on Neural Networks, vol. 8, no. 5, pp. 1131-1148, 1997.

....hidden units. The pruning process is terminated when no unit can be removed without causing the network accuracy on the cross validation set to drop below the prescribed level. While there are several papers that propose algorithms for constructing and or training neural network for regression [6, 8, 9], we have been unable to find a paper that compares the accuracy of neural networks for regression against those of other traditional methods such as statistical regression method. A recent study by Frank et al. 5] on the application of naive Bayes methodology for regression provides us with an ....

Kwok, T.Y. and Yeung, D.Y. (1997) Objective functions for training new hidden units in constructive neural networks. IEEE Trans. on Neural Networks, 8 (5) 11311148.

....unit, and the usual (squared) error criterion is used. In the cascade correlation architecture [18] and its variant [91, 104] the new hidden unit maximizes a correlation function between the residual error and the hidden unit activation. Some other correlation based functions are proposed in [11, 30, 53]. The convergence property of the cascade correlation learning procedure, for networks using the hyperbolic tangent as hidden unit transfer function and assuming the input environment measure to be uniform, is proved in [16] More general results, extending to other hidden unit transfer ....

.... of the cascade correlation learning procedure, for networks using the hyperbolic tangent as hidden unit transfer function and assuming the input environment measure to be uniform, is proved in [16] More general results, extending to other hidden unit transfer functions, correlation functions in [30, 53] and input environment measures, are discussed in [53] However, for the criterion function in [11] its convergence property is not known. Besides, a projection index [105] which finds interesting projections that deviate from Gaussian distributions, can be used. However, this criterion is ....

[Article contains additional citation context not shown here]

T.Y. Kwok and D.Y. Yeung. Objective functions for training new hidden units in constructive neural networks, 1995. Submitted.

No context found.

Kwok, T.Y., & Yeung, D.Y. (1994). Objective functions for training new hidden units in constructive neural networks. Submitted to Neural Networks.

....pattern p, E po is the residual error at output o for pattern p before the new hidden unit is added, and H and E o are the corresponding values averaged over all patterns. Some other correlation based functions, with different time and space requirements, are proposed in [65] 118] 119] [120]. The method mentioned in Section IV B.2 may also be used. An experimental comparison of the performance of these alternatives can be found in [120] An exception is [110] Here, the new hidden unit is treated as an interim output unit, as in GMDH to be discussed in Section IV E, and the error ....

....averaged over all patterns. Some other correlation based functions, with different time and space requirements, are proposed in [65] 118] 119] 120] The method mentioned in Section IV B.2 may also be used. An experimental comparison of the performance of these alternatives can be found in [120]. An exception is [110] Here, the new hidden unit is treated as an interim output unit, as in GMDH to be discussed in Section IV E, and the error criterion is directly used to train the weights. C.3 Convergence Property Except for the variant in [111] the universal approximation capability for ....

[Article contains additional citation context not shown here]

T.Y. Kwok and D.Y. Yeung, "Objective functions for training new hidden units in constructive neural networks," 1995, Submitted.

....units to the outputs are trained (output training) In so doing, only one layer of weights needs to be optimized at a time. There is never any need to back propagate the error signals and hence is much faster. A number of objective functions have been proposed for the input training phase. Some [2, 7, 8] have simple computational forms and also ensure asymptotic convergence of the procedure to the target function. Constructive procedures using these objective functions are thus particularly interesting. Regularization can be used to improve generalization performance. The basic idea is to ....

....it is still more in line with the Bayesian methodology than the evidence framework [9] because the hyperparameters are integrated out instead of being optimized. unit is added. Dropping the constant (with respect to a fixed training set) factor 1=N , we obtain the following objective function [8]: S = X p E p H p ) 2 = X p H 2 p : 3) In [8] we demonstrated experimentally that S in (3) compares favorably with other objective functions used for input training. Moreover, if we ignore the decrease in residual error as a result of output training, then, from (1) we have the ....

[Article contains additional citation context not shown here]

T.Y. Kwok and D.Y. Yeung. Objective functions for training new hidden units in constructive neural networks, 1995. Submitted.

No context found.

T.-Y. Kwok and D.-Y Yeung, "Objective Functions for Training New Hidden Units in Constructive Neural Networks," IEEE Trans. on Neural Networks, vol. 8, no. 5, Sept. 1997.

No context found.

T. Kwok, D. Yeung, "Objective functions for training new hidden units in constructive neural networks", IEEE Transactions on neural networks, vol. 8, no. 5, pp. 1131-1148, 1999.

No context found.

T. Y. Kwok and D. Y. Yeung, "Objective functions for training new hidden units in constructive neural networks," IEEE Trans. Neural Networks, vol. 8, pp. 1131--1148, Sept. 1997.

Online articles have much greater impact   More about CiteSeer.IST   Add search form to your site   Submit documents   Feedback  

CiteSeer.IST - Copyright Penn State and NEC