R. Setiono, L.C.K. Hui, Use of a quasi-Newton method in a feedforward neural network construction algorithm, IEEE Transactions on Neural Networks 6 (1995) 273--277.  Home/Search   Document Not in Database   Summary   Related Articles   Check

....obtained. The results obtained by EPNet are quite competitive in comparison with those obtained by other algorithms. Table III compares EPNet s best results with those of cascadecorrelation algorithm (CCA) 5] the perceptron cascade algorithm (PCA) 7] the tower algorithm (TA) 6] and the FNNCA [8]. All these algorithms except for the FNNCA can produce networks with short cut connections. Two observations can be made from this table. First, EPNet can evolve very compact networks. In fact, it generated the smallest ANN among the five algorithms compared here. Second, the size of the network ....

....to finish. Some of them took less than 50 generations to finish. In those cases, the average number of connections and accuracy between the last generation and the 50th one were set to be the same as those at the last generation in order to draw this figure. TABLE VIII COMPARISON AMONG FNNCA [8], A HAND DESIGNED ANN [36] AND EPNet ON THE BREAST CANCER PROBLEM. ANN s DESIGNED MANUALLY AND BY FNNCA HAVE MORE CONNECTIONS THAN THOSE EVOLVED BY EPNet, EVEN WHEN THE NUMBER OF HIDDEN NODES IS THE SAME SINCE EPNet CAN GENERATE SPARSELY CONNECTED ANN s. ONLY THE AVERAGE RESULTS FROM EPNet ARE ....

[Article contains additional citation context not shown here]

R. Setiono and L. C. K. Hui, "Use of a quasinewton method in a feedforward neural-network construction algorithm," IEEE Trans. Neural Networks, vol. 6, pp. 273--277, 1995.

....of the BKS combination with the works on Australian credit approval data set by Michie [10] The BKS combination method shows low error rate than other works do. Especially, speciation with average output shows best performance. Table 5 shows comparisons with other representative works, FNNCA [13], HDANNS [12] and Table 3: BKS combination method : Subscript A=Australian credit approval; B=Breast Cancer; D=Diabetes. Avg Entropy Pearson Average 0.9122 0.9041 0.9076 Average 0.98745 0.98916 0.98745 Average 0.7823 0.7979 0.7948 StdDev 0.0108 0.0151 0.0151 StdDev 0.006489 0.007836 ....

R. Setino and L.C.K. Hui, "Use of a quasi-newton method in a feed forward neural network construction algorithm, " IEEE Trans. on Neural Networks, vol. 6, no. 1, pp. 273-277, 1995.

....Cascade Correlation and adaptive RBF networks have been successfully used in a wide range of applications and are now established techniques. Apart from the algorithms mentioned here there are many more recent algorithms for dynamically creating or splitting nodes in single or multiple layers [65, 129, 63, 7, 102, 79, 5, 115, 123, 124]. The claimed performances are often good, at least on artificial datasets. However, node creation and splitting algorithms have been less successful in practical applications possibly because the new hidden nodes do not always systematically decrease errors for some (though not necessarily all) ....

R. Setiono and L. K. H. Hui. Use of a quasi - newton method in feed-forward neural network construction. IEEE Transactions on Neural Networks, 6:273--277, 1995.

.... they can produce representations which are efficient [15] Constructive algorithms have been the mainstay in the statistical community for many years, and in recent years many of these methods have been integrated into the neural network community [16] 8] 9] 17] 18] 7] 19] 20] 21] [22], 23] 24] 25] Second, we use piecewise linear sigmoidal nodes instead of continuously differentiable logistic nodes. This changes the nature of the learning problem entirely. It becomes a combinatorial problem in the sense that the number of feasible solutions that we must search through to ....

R. Setiono and L.C.K. Hui, "Use of a quasi--newton method in a feedforward neural network construction algorithm," IEEE Transactions on Neural Networks, vol. 6, no. 1, pp. 273--277, 1995.

.... may not be well suited to function approximation problems [30] On the other hand, constructive algorithms for function approximation have been the mainstay in the statistical community for many years, and in recent years many of these methods have been integrated into the neural network community [1, 8, 9, 12, 17, 20, 29, 33, 36, 42, 43, 44, 51]. Constructive algorithms possess several potential advantages over alternative methods. 1. There is often a strong computational advantage associated with the constructive approach. Generally speaking this results from the fact that it is often more efficient to train and combine the results of ....

R. Setiono and L.C.K. Hui. Use of a quasi--newton method in a feedforward neural network construction algorithm. IEEE Transactions on Neural Networks, 6(1):273-- 277, 1995.

....In the diagram, empty circles represent input hidden output units while a black dot refers to a connection between units. 3.1.1 Simple Hidden Units There are two main categories. The first category is based on the multi layer perceptron (MLP) like the dynamic node creation network [3] and [39, 49, 66, 86, 91, 104, 106]. The hidden unit transfer function in a MLP is of the form: g(x) OE(a T x ) where OE is usually the sigmoidal function OE(z) 1= 1 e Gammaz ) The other category is based on the radial basis function network (RBFN) like the Gaussian potential function network (GPFN) 57] the ....

....units to learn. This is commonly used in many algorithms, such as PPL type algorithms [28, 43, 82, 83, 98] 7 A function space approach to analyzing the learning algorithm of RAN is developed in [46] cascade correlation architecture and its variants [18, 59, 58, 91, 104, 105] and methods in [86, 106]. Experimentally, this strategy allows the network to learn faster. Moreover, the hidden units, acting as feature detectors, are never cannibalized once built. They are available from that time on for producing outputs or more complex features. However, because the parameters of the hidden units ....

R. Setiono and L.C.K. Hui. Use of a quasi-Newton method in a feedforward neural network construction algorithm. IEEE Transactions on Neural Networks, 6(1):273--277, 1995.

....than methods modeled after Newton s method, but they are potentially of great value for very large problems. Similarly, simple gradient descent methods, though they only require O(k) space and time for each iteration, are notoriously slow when the network size is large [79] It has been argued in [80] that these scale up problems are less important in constructive algorithms, because they always start with small networks. This may be true in simple problems. But in complex problems, though the computational requirement may not be a major concern at the early stage when the network size is ....

....3 gives a taxonomy of the constructive algorithms surveyed in this paper. Details of individual categories will be discussed in the following sections, which are named after their representative algorithms. A. Dynamic Node Creation Constructive algorithms in this category [36] 64] 77] [80], 84] 88] are variants of the dynamic node creation (DNC) network proposed by Ash [84] Here, the state transition mapping is single valued. Sigmoid hidden units are added one at a time, and are always added to the same hidden layer. The whole network must be re trained completely after each ....

R. Setiono and L.C.K. Hui, "Use of a quasi-Newton method in a feedforward neural network construction algorithm," IEEE Transactions on Neural Networks, vol. 6, no. 1, pp. 273--277, 1995.

....25 runs. For the Australian credit card assessment problem, CELA s result is comparable to those obtained using evolutionary approaches [7, 8] but better than others [9] Table 5 compares the average testing error rates among different algorithms. For the breast cancer problem, Setiono and Hui [10] have recently published a new ANN constructive algorithm called FNNCA. The best testing error rate produced by FNNCA in 50 runs was 0.0145. Prechelt [6] also reported results on manually constructed ANNs. He tested a number of different ANN architectures for the breast cancer problem. The best ....

R. Setiono and L. C. K. Hui, "Use of a quasiNewton method in a feedforward neural network construction algorithm," IEEE Trans. on Neural Networks, Vol. 6, pp.273--277, 1995.

....Gaussian hidden nodes; the last two algorithms use linear threshold nodes. All networks constructed by the above algorithms use short cut connections. Using a single hidden layer, FNNCA can construct neural networks having (3, 4, 5, 5) hidden nodes respectively that solve this problem for N=4 to 7 [7]. Based on ten runs of PBLA for each value of N , the average of the best network obtained are summarized in Table 2, where number of epochs indicates the total learning epochs taken by PBLA when the best network is obtained. Figure 2 shows an optimum network obtained by PBLA for the 7 bit ....

R. Setiono and L. C. K. Hui. Use of a quasi-Newton method in a feedforward neural network construction algorithm. IEEE Trans on Neural Networks, 6(1):273-277, 1995.

....there are strong biological and engineering evidences to support that the function, i.e. the information processing capability of an ANN, is determined by its architecture. There have been many attempts to design ANN architectures automatically, such as various constructive and pruning algorithms [1, 2, 3]. Roughly speaking, a constructive algorithm starts with a minimal network (i.e. a network with a minimal number of hidden layers, nodes, and connections) and adds new layers, nodes, and connections if necessary during training, while a pruning algorithm does the opposite, deleting unnecessary ....

....16.1 1.3 48.7 Min 27 1 100 Max 87 6 280 ffl The generalisation ability of the EANNs are very good. This is illustrated by the so called inverse behaviour of errors where the validation error is lower than the training error. For comparison, the neural networks constructed using the FNNCA method [2] have an average rate of 1.95 on the testing set for the breast cancer problem, while our result is 1.376 . For the diabetes problem, Smith et al. 22] achieved an error rate of 24 on the testing set of 192 examples (576 as training data) we achieved 22.379 . For the heart disease problem, the ....

R. Setiono and L. C. K. Hui. Use of a quasi-newton method in a feedforward neural network construction algorithm. IEEE Trans. on Neural Networks, 6(1):273--277, 1995.

....connection weights and biases. 2 This paper is only concerned with connectivity and will use architecture and connectivity interchangeably. The work on evolving both connectivity and node transfer functions was reported elsewhere [4] various constructive and pruning algorithms [5] 6] 7] [8], 9] Roughly speaking, a constructive algorithm starts with a minimal network (i.e. a network with a minimal number of hidden layers, nodes, and connections) and adds new layers, nodes, and connections if necessary during training, while a pruning algorithm does the opposite, i.e. deletes ....

....obtained. The results obtained by EPNet are quite competitive in comparison with those obtained by other algorithms. Table III compares EPNet s best results with those of cascadecorrelation algorithm (CCA) 5] the perceptron cascade algorithm (PCA) 7] the tower algorithm (TA) 6] and the FNNCA [8]. All these algorithms except for the FNNCA can produce networks with short cut connections. Two observations can be made from this table. First, EPNet can evolve very compact networks. In fact, it generated the smallest ANN among the five algorithms compared here. Second, the size of the network ....

[Article contains additional citation context not shown here]

R. Setiono and L. C. K. Hui, "Use of a quasi-newton method in a feedforward neural network construction algorithm," IEEE Trans. on Neural Networks, vol. 6, no. 1, pp. 273--277, 1995.

....where the validation error is lower than the training error. For the breast cancer problem, our EANNs have the average error rate of 3.773 on the training set, of 0.595 on the validation set and of 1.376 on the testing set. For comparison, the neural networks constructed using the FNNCA method [21] have an average rate of 1.95 on the testing set. For the diabetes problem, our EANNs have the average error rate of 24.054 on the training set, of 18.854 on the validation set and of 22.379 on the testing set. No algorithm performs very well for this problem. This data set was studied by ....

R. Setiono and L. C. K. Hui. Use of a quasi-newton method in a feedforward neural network construction algorithm. IEEE Trans. on Neural Networks, 6(1):273--277, 1995.

No context found.

R. Setiono and L. C. K. Hui, "Use of a quasi-Newton method in a feedforward neural network construction algorithm," IEEE Trans. Neural Networks, vol. 6, pp. 273--277, Jan. 1995.

....accuracy and rule simplicity (as discussed in Section 1) an appropriate number of hidden units must be determined, and two general approaches have been proposed in the literature. The constructive algorithms start with a few hidden units and add more units as needed to improve network accuracy [5, 6, 7]. The destructive algorithms, on the other hand, start with a large number of hidden units and remove those that are found to be redundant [8] The number of useful input units corresponds to the number of relevant input attributes of the data. Typical algorithms usually start by assigning one ....

R. Setiono and L. C. K. Hui, \Use of a quasi-Newton method in a feedforward neural network construction algorithm," IEEE Trans. on Neural Networks, vol. 6, no. 1, pp. 273-277, 1995.

.... and training algorithms were tested with the assumption that indeed N hidden units are necessary for solving the problem [1, 2, 5, 8] A neural network construction algorithm that employs the quasi Newton method for minimizing the error function was recently proposed by Setiono and Hui [10]. This algorithm successfully built networks having less than N hidden units that were capable to solve the N bit parity problem for small values of N ranging from 4 to 8. Indeed, it follows from a result of Sontag [11] that a sufficient number of hidden units for the network is (N=2) 1 if N is ....

R. Setiono and L.C.K. Hui, Use of quasi-Newton method in a feedforward neural network construction algorithm, IEEE Transactions on Neural Networks 6 (1) (1995) 273--277.

....with a single hidden layer, architecture selection boils down to finding appropriate numbers of units in the input and hidden layers. To find an appropriate number of hidden units, constructive algorithms start with a few hidden units and add more units as needed to improve network accuracy [1, 8, 14]. Destructive algorithms, on the other hand, start with a large number of hidden units and remove those that are found to be redundant [11] The number of useful input units correspond to the number of relevant input attributes of the data. Typical algorithms usually start by assigning one input ....

Setiono, R. and Hui, L.C.K. (1995) Use of a quasi-Newton method in a feedforward neural network construction algorithm. IEEE Trans. on Neural Networks, 6 (1), 273277.

....pruned. The penalty function (12) was minimized using a variant of the quasi Newton for unconstrained minimization to speed up convergence. This method is the BFGS (Broyden Fletcher Goldfarb Shanno) algorithm which have been shown to be superior to the backpropagation method for network training [16, 19]. The parameters of the penalty function P (w; v) were set as follows: fi = 100; ffl 1 = 1; ffl 2 = 10 Gamma3 . For each network, we recorded the smallest number of connections that were present in the network when the accuracy on the training data was at least 98 . The pruning process was ....

R. Setiono and L.C.K. Hui, Use of quasi-Newton method in a feedforward neural network construction algorithm, IEEE Trans. on Neural Networks 6 (1) (1995) 273277.

....method is the need to determine the number of units in the hidden layer prior to training. To overcome this difficulty, many algorithms that construct a network dynamically have been proposed. The algorithm which generates a single hidden layer feedforward network that we have recently proposed [8] can be outlined as follows Feedforward neural network construction algorithm 1. Let h = 1 be the initial number of hidden units in the network. Set al.l initial weights in the network randomly. 2. Find a point that minimizes an error function. 3. If this solution results in a network that ....

R. Setiono and L.C.K. Hui, "Use of quasi-Newton method in a feedforward neural network construction algorithm," IEEE Transactions on Neural Networks, vol. 6, no. 1, pp. 273-277, Jan. 1995.

....as long as the network has enough units. The TBNN method provides an upper bound on the number of units required in each layer. In our experiment, the network started with 8 input units, 6 hidden units, and 3 output units, as specified by TBNN. The network was trained using a quasi Newton method [10]. After training the network, irrelevant and redundant network connections were removed by applying the Neural Network Pruning with Penalty Function (N2P2F) algorithm, which we proposed earlier and has been shown to be very effective [9] The resulting network is shown in Fig. 4. It has only 2 ....

R. Setiono and L. C. K. Hui, Use of quasi-Newton method in a feedforward neural network construction algorithm, IEEE Trans. on Neural Networks, 6(1):273--277, 1995.

No context found.

R. Setiono, L.C.K. Hui, Use of a quasi-Newton method in a feedforward neural network construction algorithm, IEEE Transactions on Neural Networks 6 (1995) 273--277.

No context found.

R. Setiono and L. C. K. Hui, "Use of quasi-Newton method in a feed forward neural network construction algorithm," IEEE Trans. Neural Networks, vol. 6, pp. 273--277, Mar. 1995.

No context found.

L. C. K. Hui R. Setiono. Use of a quasinewton method in a feedforward neuralnetwork construction algorithm. IEEE Transactions Neural Networks, 6:273--277, 1995. 52

No context found.

R. Setiono and L. C. K. Hui, "Use of a quasi-Newton method in a feedforward neural network construction algorithm," IEEE Trans. on Neural Networks, Vol. 6, pp.273--277, 1995.

Online articles have much greater impact   More about CiteSeer.IST   Add search form to your site   Submit documents   Feedback  

CiteSeer.IST - Copyright Penn State and NEC