Kevin J. Lang and Michael J. Witbrock. Learning to tell two spirals apart. In David S. Touretzky, Geoff E. Hinton, and Terrence J. Sejnowski, editors, Proceedings of the 1988 Connectionist Models Summer School. Morgan Kauffman, 1988.  Home/Search   Document Not in Database   Summary   Related Articles   Check

....effectiveness of new learning algorithms, or proposed modifications of BP. It consists in identifying the points of two interlocking spirals. The training set comprises 194 points. Standard BP with a simple architecture has not found a solution to this problem (see [Fahlman and Labiere, 1990] [Lang and Witbrock, 1988] solved it in 20,000 epochs using standard BP with a complex architecture (2 5 5 5 1 with shortcuts) Supersab needed an average of 3,500 epochs, Quickprop 8,000, RPROP 6,000, and Cascade Correlation 1,700. To solve this problem we constructed a network with architecture 2 16 1, using the ....

Lang, K.J. & Witbrock, M.J. Learning to Tell Two Spirals Apart. Proc. 1988 Connectionist Models Summer School, Morgan Kaufmann.

....extracted. Therefore, the objective TABLE I CIRCLE IN THE SQUARE PROBLEM will be to test the capabilities of each architecture to reduce category proliferation, while preserving generalization. The first set of benchmarks will consist of variations of the well known circle in the square problem [17] that has been widely used in ARTMAP literature [6] 9] 10] It will serve to illustrate the concept of populated exception and its effect on the training of the evaluated networks. In addition, the influence of the dimensionality of the input space will be assessed on a variation of this ....

K. J. Lang and M. J. Witbrock, "Learning to tell two spirals apart," in Proc. 1988.

....partitioning (FKP) and pseudo FKP [1] to perform the cluster analysis. Such clustering techniques require prior knowledge such as the number of clusters present in a data set and are not sufficiently flexible to handle nonpartitionable problems such as the XOR dilemma and the 2 spiral problem [16]. Generally, neural fuzzy networks that employ partition based clustering techniques also lack 1045 9227 02 17.00 2002 IEEE Fig. 1. Structure of GenSoFNN. the flexibility to incorporate new clusters of data after the training has completed. This is known as the stability plasticity dilemma ....

....of the data sets and the objectives of the simulations are given in the respective sections. For all the simulations, the parameters specified in Table I are used. A. 2 Spiral Classification The 2 spiral classification problem is a complex neural network benchmark task developed by Lang [16]. The task involves learning to correctly classify the points of two intertwined spirals (denoted here as Class 0 and Class 1 spirals, respectively) The two spirals each make three complete turns in a two dimensional (2 D) plane, with 32 points per turn plus an endpoint, totaling 97 points per ....

[Article contains additional citation context not shown here]

K. J. Lang and M. J. Witbrock, "Learning to tell two spirals apart," in Proc. 1988.

....of 441 points in [ 1, 1] 1, 1] otherwise no values were reported in the corresponding columns. It was also considered the situation after 1000 learning cycles. In order to evaluate the generalization capability, another experiment was been carried out with the classical two spirals problem [11]. It is an extremely hard two class problem for plain MLP s. The task is to train on the 194 I O pairs until the learning system can produce the correct output for all of the inputs and then calculate the response for different inputs. Result of tests performed on 10200 equally spaced points in ....

Lang K. and Witbrock, "Learning to tell two spiral apart", in Proc. 1988.

....that, for every learning method, an important practical aspect is the number of pattern presentations necessary to achieve a satisfying performance. Fritzke says that GCS needs 180 epochs to achieve the above mentioned result and reports comparisons with some earlier methods: Backpropagation [54] (20000 epochs) Cross Entropy BP [54] 10000 epochs) Cascade Correlation [55] 1700 epochs) and he says that the number of epochs required by GCS is about one or two orders of magnitude less than the other techniques reported. We can see that FACS needs about 15 iterations to obtain the same ....

....an important practical aspect is the number of pattern presentations necessary to achieve a satisfying performance. Fritzke says that GCS needs 180 epochs to achieve the above mentioned result and reports comparisons with some earlier methods: Backpropagation [54] 20000 epochs) Cross Entropy BP [54] (10000 epochs) Cascade Correlation [55] 1700 epochs) and he says that the number of epochs required by GCS is about one or two orders of magnitude less than the other techniques reported. We can see that FACS needs about 15 iterations to obtain the same result as the GCS, i.e. a twelfth of the ....

K. J. Lang and M. J. Witbrock, \Learning to tell two spirals apart," in Proceedings of the 1988.

....0.6 0.8 1 Figure 6.13: Two spirals: supervised classi cation. e T = 0:001; N C = 144 presentations necessary to achieve a satisfying performance. Fritzke says that GCS needs 180 epochs to achieve the above mentioned result and reports comparisons with some earlier methods: Backpropagation [93] (20000 epochs) Cross Entropy BP [93] 10000 epochs) Cascade Correlation [94] 1700 epochs) and he says that the number of epochs required by GCS is about one or two orders of magnitude less than the other techniques reported. We can see that FACS needs about 15 iterations to obtain the same ....

....supervised classi cation. e T = 0:001; N C = 144 presentations necessary to achieve a satisfying performance. Fritzke says that GCS needs 180 epochs to achieve the above mentioned result and reports comparisons with some earlier methods: Backpropagation [93] 20000 epochs) Cross Entropy BP [93] (10000 epochs) Cascade Correlation [94] 1700 epochs) and he says that the number of epochs required by GCS is about one or two orders of magnitude less than the other techniques reported. We can see that FACS needs about 15 iterations to obtain the same result as the GCS, i.e. a twelfth of the ....

K. J. Lang and M. J. Witbrock, \Learning to tell two spirals apart," in Proceedings of the 1988.

....I . Fuzzy ARTMAP . s0 f 75 . SuperSAB 1 70 , I I I 100 200 500 1000 mining set size Fig. 4. Comparison of test set accuracy of circle in the square 4. 3 Telling Two Spirals Apart Telling two spirals apart is a neural network benchmark test proposed by Wieland [17]. Each of the two spirals of this task makes three complete rams in the plane, with 32 points per mm plus an endpoint. Thus, there are 194 training instances in sum. The training instances (x , y, b ) k = 1, 2 . 194) are generated according to the following equations, where n l, 2 . ....

....y c [0, 1] in steps of 0.01 means it is a white point, and b k black point. x 2 =r sinc 0.5=l x x 2 = 0.5 r sina; l x 2q co; b z = 1 b 2n = 0 04(105 n r: 0 means it is a x(n 1) con (Equation 21) 16 This benchmark test is a quite difficult task. Lang and Witbrock [17] reported that it is unable to accomplish this task using a standard BP network with connections from one layer to the next. In order to solve this problem, they crafted a special 2 5 5 5 1 architecture with shortcut connections, each node being connected to all nodes in all subsequent layers. ....

[Article contains additional citation context not shown here]

K.J. Lang, M. J. Witbrock. Learning to Tell Two Spirals Apart. In: Proc. 1988 Connectionist Models Summer School, Morgan Kaufmann: San Mateo, CA, 1989, pp.52-59.

....from the points in the learning set by 0. 1. The best solutions to the spirals problem have been obtained using Cascade Correlation [11] This is capable of approximating the problem in 1700 epochs using around 140 parameters. Back propagation networks have been used to solve the problem [12] using a 2 5 5 1 network with shortcut connections between layers in 20, 000 epochs using conventional gradient descent with momentum and 8, 000 epochs using quickprop, using a similar number of parameters. Using a conventional 2 5 5 1 network without shortcuts took 60, 000 epochs of quickprop. ....

K. J. Lang and M. J. Witbrock, "Learning to tell two spirals apart," in Proceedings of the 1988.

....the spiral problem has received much attention from several different fields. In the neural network community, a variant of the problem has become a benchmark [10] to evaluate the performance of a learning system on both nonlinear separability and generalization, since Lang and Witbrock [25] reported that the problem could not be solved with a standard multilayered perceptron. Many papers have dealt with the problem and many solutions have been attempted using different learning Manuscript received November 23, 1999; revised November 29, 2000. This work was supported in part by NSF ....

....University, Columbus, OH 43210 1277 USA (e mail: dwang cis.ohio state.edu) Publisher Item Identifier S 1045 9227(01)06459 1. a) b) Fig. 1. The spiral problem. a) A connected single spiral. b) Disconnected double spirals (adapted from [29] and [30] models [2] 5] 7] 11] 13] [25]. However, resulting learning systems are only able to produce decision regions highly constrained by the spirals defined in a training set, thus specific in shape, position, size, orientation, etc. The problem in the geometric form is still open for neural network learning. Furthermore, no ....

K. Lang and M. Witbrock, "Learning to tell two spirals apart," in Proc.

....from the points in the learning set by 0. 1. The best solutions to the spirals problem have been obtained using Cascade Correlation [11] This is capable of approximating the problem in 1700 epochs using around 140 parameters. Back propagation networks have been used to solve the problem [12] using a 2 5 5 1 network with shortcut connections between layers in 20, 000 epochs using conventional gradient descent with momentum and 8, 000 epochs using quickprop, using a similar number of parameters. Using a conventional 2 5 5 1 network without shortcuts took 60, 000 epochs of quickprop. ....

K. J. Lang and M. J. Witbrock, "Learning to tell two spirals apart," in Proceedings of the 1988 Connectionist Models Summer School, Morgan Kaufmann, 1988.

....novel neural and statistical pattern recognition classifiers. Considerable work has been done in the area since mid 1980s and in 1990s and a number of intelligent approaches have been applied to solving the spiral problem; neural networks: Fahlman[2] Fahlman and Lebiere[3] Lang and Witbrock[4], Tay and Evans[5] neurofuzzy methods: Sun and Jang[6] and data encoding methods: Chua et al. 7] Jia and Chua[8] In addition, several other studies have tested their proposed pattern recognition methods on this benchmark problem since this process served as an indicator of their success with ....

Lang, KJ and Witbrock, MJ. Learning to tell two spirals apart, In Proceedings of the 1988 Connectionist Models Summer School, Morgan Kaufmann, 1988.

....from the fact that the bounds are constructive as quantizing IR n in a particular prescribed manner between size and the entropy bounds is very practical. The quantization used can represent the first step of a constructive learning algorithm (for a classical classification problem [24], several results are presented in Fig. 2) 4. Conclusions Based on the entropy of the data set, this paper has presented a new constructive proof on the number of bits required for solving a dichotomy problem. The resulting lower bounds are tighter than the ones previously known. V. Beiu, S. ....

K.J. Lang and M.J. Witbrock, "Learning to tell two spirals apart," in Proc. of the 1988 Connectionist Models Summer School, pp. 52-59, Morgan Kaufmann: San Mateo, CA, 1988.

....complete fault tolerance. Next, we consider some realistic benchmark problems well suited for ANNs. The training and testing data and a description of the above benchmarks was obtained from the public database maintained by Fahlman et al. at CMU [10] The Two Spirals Classification Benchmark [9, 10, 13] : The task is to learn to discriminate between two sets of training points which lie on two distinct spirals in the x y plane. The two spirals are illustrated in Figure 3. These spirals coil three times around the origin and around one another. The training data consists of two sets of points, ....

Lang, K. J., and Witbrock, M. J. "Learning to Tell Two Spirals Apart". In Proceedings of the 1988 Connectionist Models Summer School, San Mateo, CA, 1988, D. Touretzsky, G. Hinton, and T. Sejnowski, Eds., Morgan Kaufman Publishers.

....the components of the vector x. The total region belonging to a specific class is obtained by combining all the rectangles by OR gates. To obtain a limited fan in the OR gates can be cascaded. An Example: To demonstrate the method we use a noisy variant of the well known two spirals problem [5]. Two probability densities were generated by superimposing Gaussians with variance oe 2 = 0:1 centered at 485 points on each Figure 1: left) The noisy spiral: P (xjC 1 ) generated by a superposition of Gaussians with variance oe 2 = 0:1. Figure 2: right) 1000 examples drawn from P (xjC 1 ) ....

Lang K.J., and Witbrock M.J. (1988). Learning to tell two spirals apart. Proc. Connectionists Models Summer School, Morgan Kauffmann, San Mateo, 52--59.

....novel neural and statistical pattern recognition classifiers. Considerable work has been done in the area since mid 1980s and in 1990s and a number of intelligent approaches have been applied to solving the spiral problem; neural networks: Fahlman[2] Fahlman and Lebiere[3] Lang and Witbrock[4], Tay and Evans[5] neurofuzzy methods: Sun and Jang[6] and data encoding methods: Chua et al. 7] Jia and Chua[8] In addition, several other studies have tested their proposed pattern recognition methods on this benchmark problem since this process served as an indicator of their success with ....

Lang, KJ and Witbrock, MJ. Learning to tell two spirals apart, In Proceedings of the 1988 Connectionist Models Summer School, Morgan Kaufmann, 1988.

....line sweep is given by z a z b = 5. 1 1 x 4 3 2 1 x x 1 x x x x z a = 2 b z = 3 2 4 b a a b 5 Simulation Results We illustrate the working of the sparse multinomial classifier for the 2 spirals problem. In the original formulation (Lang and Witbrok 1988), the classifier has to find a decision surface to separate two continuous point sets in IR 2 that belong to one or the other of intertwined spirals. The problem is formulated for binarized inputs as follows: each point in the plane is represented by some bitstring B x B y which is the ....

Lang, K. J.; and Witbrok M. 1988. Learning to tell two spirals apart. In: Proceedings of the 1988 Connectionists Model Summer School, pp. 52-59. D. S. Touretzky, G. E. Hinton, T. J. Sejnowski, eds. New York: Morgan Kaufmann Publishers.

....they are not linearly separable. Finding a neural network solution to the two spirals problem has proven to be very difficult when using a traditional gradient descent learning method such as backpropagation, and, therefore, it has been used in a number of studies to test new learning methods (Lang and Witbrock, 1988; Fahlman and Lebiere, 1990; Whitley and Karunanithi, 1991; Suewatanakul and Himmelblau, 1992; Potter, 1992; Karunanithi et al. 1992) Figure 9: Training set for the two spirals problem. To learn to solve this task, we are given a training set consisting of 194 preclassified 3 More accurately, ....

Lang, K. J. and Witbrock, M. J. (1988). Learning to tell two spirals apart. In Touretzky, D., Hinton, G. and Sejnowski, T., editors, Proceedings of the 1988 Connectionist Models Summer School, pages 52--59, Morgan Kaufmann, San Mateo, California.

....network outputs, the sigmoid function is applied at the output units. Karnin (1990) used the hyperbolic tangent function as the activation function at the hidden units. We have also used this function to obtain all the results reported in this paper. 5 It has been suggested by several authors (Lang and Witbrock 1988; van Ooyen and Nienhuis 1992) that the cross entropy error function F (w; v) Gamma k X i=1 C X p=1 t pi log S pi (1 Gamma t pi ) log(1 Gamma S pi ) 2.3) improves the convergence of the training process. The components of the gradient of this function are F (w; v) w m = F ....

Lang, K.J., and Witbrock, M.J. 1988. Learning to tell two spirals apart. In Proc. of the 1988 Connectionist Summer School, 52--59. Morgan Kaufmann, San Mateo, CA.

.... e Gammay ) For all the results reported in this 6 paper, we have used the hyperbolic tangent function as the activation function at the hidden units and the sigmoid function at the output unit. To improve the convergence in neural network training, it has been suggested by several authors [9] [10] that the cross entropy error function be used F (w; v) Gamma k X i=1 i t i log S i (1 Gamma t i ) log(1 Gamma S i ) j : 6) We use this cross entropy function in conjunction with our construction algorithm. Given two disjoint sets of patterns, it has been shown [11] ....

K.J. Lang and M.J. Witbrock, "Learning to tell two spirals apart," in Proceedings of the 1988 Connectionist Model Summer School, edited by D. Touretzky, G. Hinton, and T. Sejnowski, Morgan Kaufmann, San Mateo, CA., pp. 52-59, 1988.

....No test set is provided. In order to solve the two spiral problem we generated a two dimensional EGCS. Figure 2 shows the EGCS1 and EGCS2 and their corresponding decision regions. Black indicates assignment to the first, white Network Model Number of epochs Reported in BP (SSE) 20000 K. J. Lang [13] BP (CE) 11000 K. J. Lang [13] Quickprop 7900 K. J. Lang [13] Cascor RPROP 2437 N. K. Treadgold [17] Cascor 1700 S. E. Fahlman [5] SGCS 180 B. Fritzke [7] DCS GCS 177 J. Bruske [3] EGCS1 104 this article EGCS2 86 this article Table 1. Epochs for Supervised Learning of the Two Spirals ....

....to solve the two spiral problem we generated a two dimensional EGCS. Figure 2 shows the EGCS1 and EGCS2 and their corresponding decision regions. Black indicates assignment to the first, white Network Model Number of epochs Reported in BP (SSE) 20000 K. J. Lang [13] BP (CE) 11000 K. J. Lang [13] Quickprop 7900 K. J. Lang [13] Cascor RPROP 2437 N. K. Treadgold [17] Cascor 1700 S. E. Fahlman [5] SGCS 180 B. Fritzke [7] DCS GCS 177 J. Bruske [3] EGCS1 104 this article EGCS2 86 this article Table 1. Epochs for Supervised Learning of the Two Spirals Problem. BP Backpropagation, ....

[Article contains additional citation context not shown here]

K. J. Lang and M. J. Witbrock. Learning to tell two spirals apart. In Proc. of the 1988 Connectionist Summer School. Morgan Kaufmann, 1988.

....results are reported in 2 Section IV. Finally, a conclusion is given in Section V. II. NEURAL NETWORK TRAINING Let us consider the standard fully connected three layer network depicted in Figure 1. The error measure that we minimize during the training process is the cross entropy function[4, 5]: F (w; v) Gamma 0 k X i=1 C X p=1 t i p log S i p (1 Gamma t i p ) log(1 Gamma S i p ) 1 A (1) Hidden Layer Input Layer w Output Layer v m m l p Figure 1: Fully connected feedforward neural network with 5 hidden units and 3 output units. 3 where ffl k is the ....

K.J. Lang and M.J. Witbrock, "Learning to tell two spirals apart", in Proc. of the 1988 Connectionist Model Summer School, D. Touretzky, G. Hinton & T. Sejnowski (Eds), Morgan Kaufmann, San Mateo, CA., 1988, pp. 52-59.

....often misclassified. For this reason, the two spiral problem has been particularly popular for testing novel neural and statistical pattern recognition classifiers. Considerable work has been done in the area since mid 1980s and in 1990s : neural networks (Fahlman, 1988; Fahlman and Lebiere, 1990; Lang and Witbrock, 1988; Tay and Evans, 1994) neurofuzzy methods (Sun and Jang, 1993) and data encoding methods (Chua et al. 1995; Jia and Chua, 1995) have been applied to solving the spiral problem. In addition, several other studies have tested their proposed pattern recognition methods on this benchmark problem ....

K. J. Lang and M. J. Witbrock, "Learning to tell two spirals apart," Proceedings of the 1988 Connectionist Models Summer School. Morgan Kaufmann, 1988.

....ideally the recognition method should be reliable with noisy data. The spiral task is difficult to solve using the backpropagation method of training neural networks. In general, large networks with complicated architectures can yield good results after long training times. Lang and Witbrock [2] solved the spiral problem using a 2x5x5x5x1 backpropagation network with learning times of 18,900 epochs, 22,300 epochs and 19,000 epochs on three different trials. By changing the error function, these authors are able to get better results with an average of 11,000 epochs over three trials. The ....

Lang, K. J. & Witbrock, M. J. (1988), Learning to tell two spirals apart. Proceedings of the 1988 Connectionist Models Summer School, Morgan Kaufmann.

.... amount of previous work exists on the subject to make valid comparisons; c) spiral data exists in several scientific applications and its recognition is of practical importance [9] and d) neural networks using backpropagation and its relatives encounter significant convergence problems [10, 11, 12] and the trained system is underconstrained [13] It has been realised in the past that recognising the 4 benchmark itself is not sufficient for a classifier: in addition, it should work in real time and be resistant to random noise variations. The latter aspect has not been studied in detail as ....

K. J. Lang and M. J. Witbrock, "Learning to tell two spirals apart," Proceedings of the 1988 Connectionist models summer school, Morgan Kaufmann, (1988).

....multidimensional input space in a similar way to that in the last section, but the decomposition occurs based on old Worker weight information. This represents a compromise between incremental learning and batch decomposition. 4. 1 The two spirals problem The two spirals problem is defined in [28] by the 192 points making up the learning set. This is shown graphically in Figure 7a. The black points are to be mapped to 1 and the white points to 0. For testing 4900 points are chosen regularly from the space and the output of the system rounded to 0 if the output lies in [0; 0:4] to 1 if the ....

K. Lang and M. Witbrock. Learning to tell two spirals apart. In D. Touretzky, editor, Proceedings of the 1988 Connectionist Summer School, pages 52--59. Morgan Kaufmann, 1988.

....novel neural and statistical pattern recognition classifiers. Considerable work has been done in the area since mid 1980s and in 1990s and a number of intelligent approaches have been applied to solving the spiral problem; neural networks: Fahlman[2] Fahlman and Lebiere[3] Lang and Witbrock[4], Tay and Evans[5] neurofuzzy methods: Sun and Jang[6] and data encoding methods: Chua et al. 7] Jia and Chua[8] In addition, several other studies have tested their proposed pattern recognition methods on this benchmark problem since this process served as an indicator of their success with ....

K. J. Lang and M. J. Witbrock, Learning to tell two spirals apart, In Proceedings of the 1988 Connectionist Models Summer School, Morgan Kaufmann, (1988).

....applied to solving the spiral problem; neural networks: Singh, S. Neural Learning of Spiral Structures , Proc. International Conference on Advances in Pattern Recognition (ICAPR 98) Plymouth, UK, Springer, pp. 226 231 (23 25 November, 1998) Fahlman[2] Fahlman and Lebiere[3] Lang and Witbrock[4], Tay and Evans[5] neurofuzzy methods: Sun and Jang[6] and data encoding methods: Chua et al. 7] Jia and Chua[8] In addition, several other studies have tested their proposed pattern recognition methods on this benchmark problem since this process served as an indicator of their success with ....

Lang, KJ and Witbrock, MJ. Learning to tell two spirals apart, In Proceedings of the 1988 Connectionist Models Summer School, Morgan Kaufmann, 1988.

....class data lies on two distinct spirals that coils around each other and the origin. Neural networks using backpropagation and its relatives encounter significant problems with the spiral problem [16] Neural solutions have been found to be too slow and are only suited to off line training [4, 5, 9]. For this reason, they are not well suited for real time spiral and other temporal data analysis. In this vein, recognising the importance of training and classifying data in real time, previous attempts on the spiral task include minimal configuration neural nets [2] input data encoding methods ....

K. J. Lang and M. J. Witbrock, "Learning to tell two spirals apart," Proceedings of the 1988 Connectionist models summer school, Morgan Kaufmann, 1988.

....spirals, each of which rotates around the origin three times. Previous results by other researchers for this problem have mostly focused on traditional MLPs and MLPs with shortcut connections. The best reported results for 100 classification accuracy are summarized below: Lang Witbrock [9]) 2 5 5 5 1 MLP with shortcut connections and 138 total weights. Average convergence time of 20,000 batched backpropagation epochs. Lang Witbrock [9] 2 5 5 5 1 MLP with shortcut connections, 138 total weights, and cross entropy error function. Average convergence time of 12,000 batched ....

....and MLPs with shortcut connections. The best reported results for 100 classification accuracy are summarized below: Lang Witbrock [9] 2 5 5 5 1 MLP with shortcut connections and 138 total weights. Average convergence time of 20,000 batched backpropagation epochs. Lang Witbrock [9]) 2 5 5 5 1 MLP with shortcut connections, 138 total weights, and cross entropy error function. Average convergence time of 12,000 batched backpropagation epochs. Lang Witbrock [9] 2 5 5 5 1 MLP with shortcut connections and 138 total weights. Average of 7,900 quickprop [3] epochs. ....

[Article contains additional citation context not shown here]

Kevin J. Lang and Michael J. Witbrock. Learning to tell two spirals apart. In Proceedings of the 1988 Connectionist Models Summer School. Morgan Kaufmann, 1988.

....required to try and solve the problem in n dimension, where n 2. Currently, linear statistical approaches to classifying two spiral data culminate in decision making by chance (50 success) Neural network solutions are tedious and require several attempts at arriving at an optimal architecture [6,7,10]. Some previous work has attempted to solve the above problem in real time by either: data encoding [5,8] using novel algorithms, hypercube separation algorithm [20] neurofuzzy systems [17] and novel neural network architectures [4,18] The solutions are however deficient because: 1) test ....

.... ) and 2 (X , Y , Z ) 3 (X , Y , Z ) then test pattern (X , Y , Z ) C 2 If 3 (X , Y , Z ) 1 (X , Y , Z ) and 3 (X , Y , Z ) 2 (X , Y , Z ) then test pattern (X , Y , Z ) C 3 [9] Go to step 3 for the next test pattern analysis. [10] Calculate the total number of test patterns correctly predicted belonging to their class. The recognition rate R is the proportion to the total number of correctly predicted patterns to the total size of the test space, i.e. the number of patterns in . The conditional memberships 1 (Y ) ....

K. J. Lang and M. J. Witbrock, "Learning to tell two spirals apart," Proceedings of the 1988 Connectionist models summer school, Morgan Kaufmann, 1988.

....for these benchmarks, whatever the learning method be used [13, 1] Learning error Generalization error Breiman wave forms 7.7 17.2 Sonar discrimination 1 7. 7 Table 3: Results for the regularized JNN learning algorithm The learning algorithm has been tested on the two spiral problem also [19]. Figure (4) shows the outputs, thresholded by an arctan function, for two networks computed with the plain and the regularized JNN algorithms respectively. Figure (5) shows the mapping of a network computed by the regularized algorithm, with a binary threshold output function. The stars represent ....

.... 2 Gamma 0:9) 2 ) ffl Complicated Interaction Function f 5 (x 1 ; x 2 ) 1:9 Gamma 1:35 e x1 sin Gamma 13(x 1 Gamma 0:6) 2 Delta e Gammax 2 sin(7x 2 ) Delta All these functions have been scaled and translated to have a standard deviation of 1 and a nonnegative range [19]. The learning set contains 225 examples randomly chosen in [0; 1] 2 . In the five examples, the targets for (d p1 ; d p2 ) are f i (d p1 ; d p2 ) 0:25n p where n p is an i.i.d. N(0; 1) noise. Five learning set have been created, one for each function, and the algorithm has been run one time ....

K. Lang and M. Witbrock. Learning to tell two spirals apart. In M. Kaufmann, editor, 1988 Connectionnist Models Summer School, 1988.

....is a language and system for symbolic and numeric mathematical calculation. It is copyrighted by the University of Waterloo, the authors of Maple, and Waterloo Maple Software. Information can be obtained from the URL: http: daisy.uwaterloo.ca The fifth problem is the well known two spirals [7] problem. Two sets of vectors from [0; 1] 2 whose positions have the form of two revolving spirals belong to the classes 0 and 1. We used 770 training vectors and then the initial AFNT of 70 decision nodes increased its performance measured by the required 40 20 40 criterion [3] within 45 epochs ....

K. J. Lang and M. J. Witbrock. Learning to tell two spirals apart. In Proc. of the 1988 Connectionist Models Summer School. Morgan Kaufmann, 1988.

....field of selective sampling becoming especially relevant in modern data mining applications. 8. Acknowledgments The authors would like to thank Neri Merhav for the helpful discussions. 35 Notes 1. We used a code available from [http: www.boltz.cs. cmu.edu benchmarks two spirals.html] and (Lang Witbrock, 1988) as a basis for our two spirals data generation program. ....

Lang, K. J. & Witbrock, M. J. (1988). Learning to tell two spirals apart. In Proceedings of the Connectionist Models Summer School, (pp. 52--9). Morgan Kaufmann.

....These two intertwined spirals are shown as Theta and ffi in Figure 2.4. This learning problem, originated by Alexis Wieland, has been a challenge for pattern classification algorithms and has been the subject of much work in the AI community, in particular in the neural network field (e.g. [55, 26, 15]) In neural network classification systems based on linear, quasi linear, radial, or clustering basis functions, the intertwined spirals problem leads to difficulty. When it is solved, the neural net solution often has a very expansive description of the spiral, i.e. the conjunction of many ....

Kevin J. Lang and Michael J. Witbrock. Learning to tell two spirals apart. In Proceedings of the 1988 Connectionist Summer Schools. Morgan Kaufmann, 1988.

....samples each were used. algorithm performance #units #epochs MLP (RPROP) 90.4 50 250 PNN 91.3 104 RBF 90.7 80 150 RCE 77.9 68 3 P RCE 90.4 68 3 DDA RBF 93.3 68 3 ffl Two Spirals: This well known problem is often used to demonstrate the generalization capability of a network (see [5]) The required task involves discriminating between two intertwined spirals. For this paper the spirals were changed slightly to make the problem more demanding. The original spirals radius declines linearly and can be correctly classified by RBF networks with one global radius. To demonstrate ....

K. Lang, M. Witbrock: "Learning to Tell Two Spirals Apart", in Proc. of Connectionist Models Summer School, 1988.

....of 194 patterns, describing the points of two distinct spirals in the x y Plane. The network is built up of 2 input units, three hidden layers with 5 units each, and 1 output unit with symmetric activation functions. Each unit is connected to every unit in earlier layers (short cut connections [7]) The results reported so far are an average of 20.000 epochs on three( different runs for Backpropagation (using both an increasing learnig rate and momentumfactor) and an average of 11.000 epochs using a nonlinear cross entropy error function. Using Quickprop, an average of 7900 epochs on ....

K. Lang and M. Witbrock. Learning to tell two spirals apart. In Proceedings of 1988 Connectionist Models Summer School. Morgan Kaufmann, 1988.

....leads to improved ensemble performance. of the problem) It appears to be extremely hard for back propagation networks due to its high non linearity. It is easy to see that the 2D points of the spirals could not be separated by small combination of linear separators. Lang and Witbrock [22] proposed a 2 Gamma 5 Gamma 5 Gamma 5 Gamma 1 network with short cuts using 138 weights. They used a variant of the quick prop learning algorithm [10] with weight decay. They claimed that the problem could not be solved with simpler architecture (i.e. less layers or without short cuts) ....

....that similar conclusions can be obtained when using a highly flexible generalized additive model (GAM) 16] Acknowledgements Stimulating discussions with Leo Breiman, Brian Ripley and Chris Bishop are greatfully acknowledged. Appendix A The Spiral data The two dimensional spiral data 10 [22] is given by a vector (x i ; y i ) defined by: x i = r i cos(ff i k =2) y i = r i sin(ff i k =2) 4) where ff i = i=16; r i = 0:5 i=16; i = 0; 97 (5) and k = 1 for one class and 3 for the other class. B Details and preprocessing of the Cleveland Heart data The data in the UCI ....

K. J. Lang and M. J. Witbrock. Learning to tell two spirals apart. In D. S. Touretzky, J. L. Ellman, T. J. Sejnowski, and G. E. Hinton, editors, Proceedings of the 1988 Connectionists Models, pages 52--59. 1988.

....100 150 200 250 300 350 400 450 500 40 45 50 55 60 65 70 75 Generations 2 Spirals, 3 Revolutions, Experiment II Learning Generalizing Figure 5 7: Two spiral, 1 revolution problem, experiment III 5.3. 5 Other Approaches to the Spiral Problem Back propagation A very good result has been reported in [LW88], where a 2 5 5 5 1 feed forward network has been used (i.e. 2 inputs and three hidden layers, with 5 neurons each) Each unit receives input not only from the units of the previous level, but from units of all the previous levels. The neural network was able to solve the task employing the ....

K. J. Lang and M. J. Witbrock. Learning to tell two spirals apart. In Proc. of the 1988 Connectionist Summer School. Morgan Kaufmann, 1988.

....18 sec. on average) Total number of games per second 8192 4915 2730 (on average) problem, originated by Alexis Wieland, perhaps based on the cover of Perceptrons, has been a challenge for pattern classification algorithms, and has been the subject of much work in the Neural Network field (e.g. [14, 7, 6]) It consists of classifying points into two classes according to two intertwined spirals. The data set is composed of two sets of of 97 points, on the plane between 7 and 7. Koza ( 11] and Angeline s chapter ( 5] also investigate this problem using the Genetic Programming paradigm. Basically, ....

Kevin J. Lang and Michael J. Witbrock. Learning to tell two spirals apart. In Proceedings of the 1988 Connectionist Summer Schools, Morgan Kaufmann.

....approach to the comparison between these five methods. 4.2 Simulation results We focused our simulation on two classification problems previously used in benchmark tests. In both problems input vectors should be classified to one of two classes. The first is the well known two spirals problem [3] which is designed to be hard to learn and the second is a real life problem concerning breastcancer diagnosis [5] Algorithm Run time Tr.err. Standard back propagation 45 min. 10 Construction method 1 45 min. 46 Construction method 2 45 min. 40 Construction method 3 45 min. 44 Construction ....

K. J. Lang and M. J. Withbrock. Learning to tell two spirals apart. In D. Touretzky, editor, Proceedings, 1988 Connectionist Models Summer School, volume 1, pages 52--59. Morgan Kaufmann, San Mateo, 1988.

....points on the plane into two classes according to two intertwined spirals. This learning problem, originated by Alexis Wieland, has been a challenge for pattern classification algorithms and has been subject of much work in the AI community, in particular in the Neural Network field (e.g. [14, 6, 3]) The data set is composed of two sets of 97 points, on the plane between 7 and 7. These two intertwined spirals are shown as Theta and ffi in figure 2, along with an example of a solution to the problem discovered in one of our experiments. Experimental Setup: We used the Genetic ....

Kevin J. Lang and Michael J. Witbrock. Learning to tell two spirals apart. In Proceedings of the 1988 Connectionist Summer Schools. Morgan Kaufmann, 1988.

....from the field of neural networks. This learning problem, originated by Alexis Wieland, perhaps based on the cover of Perceptrons, has been a challenge for pattern classification algorithms and has been subject of much work in the AI community, in particular in the Neural Network field (e.g. [Lang Witbrock, 1988, Fahlman Lebiere, 1990, Carpenter et al. 1992] In Neural Network classification systems, based on linear, quasi linear, radial, or clustering basis function, the intertwined spirals problem leads to difficulty. When it is solved, the neural net solution often has a very expansive ....

Lang, K. J. & Witbrock, M. J. (1988). Learning to tell two spirals apart. In Proceedings of the 1988 Connectionist Summer Schools. Morgan Kaufmann.

....was not noticed. Pooled update also allows one to store successive approximations to the slope of the weight space with respect to the training set, permitting the use of the Quickprop weight update rule. This rule converges considerably more quickly that the usual update rule for many problems [5, 10]. Finally, the apparent disadvantage of pooled update is reduced when observes that update doesn t have to be pooled over all n p patterns. In fact, weights can be updated after one case has been processed on each processor. 4. Simulator Implementation 4.1. Parallelizing Backprop for GF11 There ....

....21 10.3. Improving Performance for Small Networks In general, our code is optimized for networks with large enough layers to fill up the 25 cycle pipeline. However, some neural network architectures for large applications have small layers, e.g. classification tasks with a few output units[10]. Speeding up computation on smaller layers requires minimizing the longest chain of dependent operations so that more operations can be interleaved. In the forward pass, 2 floating point operations per weight are executed. Multiplies are independent of each other, but all the additions to the ....

Lang, K.J. and Witbrock, M.J., "Learning to Tell Two Spirals Apart", Proceedings of the 1988 Connectionist Models Summer School, Morgan Kaufman, San Mateo. 1988.

No context found.

Kevin J. Lang and Michael J. Witbrock. Learning to tell two spirals apart. In David S. Touretzky, Geoff E. Hinton, and Terrence J. Sejnowski, editors, Proceedings of the 1988 Connectionist Models Summer School. Morgan Kauffman, 1988.

No context found.

K. J. Lang and M. J. Witbrock (1988), Learning to tell two spirals apart, in "Proceedings of the 1988 Connectionist Summer School", Morgan Kaufman.

No context found.

K. Lang and M. Witbrock. Learning to tell two spirals apart. Proceedings of the connectionist Models summer School, pages 52--59, 1988.

No context found.

K. J. Lang and M. J. Witbrock. Learning to tell two spirals apart. In T. Sejnowski D. Touretzky, G. Hinton, editor, Proceedings of the 1988.

No context found.

K. Lang and M. Witbrock. Learning to tell two spirals apart. In M. Kaufmann, editor, 1988.

No context found.

K. J. Lang, M. J. Witbrock. Learning to Tell Two Spirals Apart, in Proceedings of the 1988.

No context found.

Lang, K., Witbrock, M.,: "Learning to tell two spirals apart", Proceedings of the 1988 Connectionist Models Summer School, M. Kaufmann Publ., 1988.

First 50 documents  Next 50

Online articles have much greater impact   More about CiteSeer.IST   Add search form to your site   Submit documents   Feedback  

CiteSeer.IST - Copyright Penn State and NEC