Parberry, I. A primer on the complexity theory of neural networks. In Formal Techniques in Artificial Intelligence: A Sourcebook, editor R. B. Banerji, 217--268, Elsevier, North-Holland, Amsterdam, 1990.  Home/Search   Document Not in Database   Summary   Related Articles   Check

....Hopfield nets is the MIN ENERGY or GROUND STATE problem of finding a network state with minimal energy (4) for a given symmetric neural network. Remember that in (4) it is assumed, for reasons of simplicity, that w jj = 0 and w j0 = 0 for j = 1; n. In addition, without loss of generality [21], we will work throughout this section with frequently used bipolar states Gamma1; 1 of neurons instead of binary ones 0; 1 introduced in (2) where 0 is now replaced by Gamma1. This problem appears to be of a special interest since many hard combinatorial optimization problems have been ....

Parberry, I. A primer on the complexity theory of neural networks. In Formal Techniques in Artificial Intelligence: A Sourcebook, editor R. B. Banerji, 217--268, Elsevier, North-Holland, Amsterdam, 1990.

....threshold circuits of size S(n) TC 0 denotes TC 0 (n O(1) TC 0 captures the complexity of important natural computational problems such as sorting, counting, and integer multiplication. It is also a good complexity theoretic model for the neural net model of computation (see [Par90] It is easy to observe that ACC 0 # TC 0 (for example, see [BIS90] and thus we have even fewer lower bounds for the threshold circuit model than for ACC 0 circuits. Furthermore, since TC 0 (s(n) # DSPACE(log s(n) for s(n) # n) and since (by an easy consequence of the space ....

I. Parberry. A primer on the complexity theory of neural networks. In R. Banerji, editor, Formal Techniques in Artificial Intelligence: A Sourcebook, volume 6 of Studies in Computer Science and Artificial Intelligence, pages 217--268. North-Holland, Amsterdam, 1990.

....and continuous time to point out alternative sources of efficient computation. 1 Introduction The computational potential and limits of neural networks have been studied for more than a decade in order to understand what is, either ultimately or efficiently, computable by particular models [15, 28, 48, 50, 53, 54, 55, 56, 58, 61, 67, 72]. This interest is motivated partly by the quest to formally justify heuristics used in practical neurocomputing, and partly by the realization that despite their formal simplicity, neural networks are computationally quite powerful, and thus may serve as a useful reference model for investigating ....

.... are necessary [29] and sufficient [49] for representing a single integer weight parameter is (s log s) Sometimes, when more appropriate, bipolar values f1; 1g (or even more general discrete domains) can be substituted for binary values f0; 1g without any substantial change in the size of weights [55, 56]. On the other hand, analog state networks usually approximate discrete activation function (4) with some continuous sigmoid function. For simplicity, we will mostly fix the activation function to be the saturated linear map: 8 : 1 for 1 for 0 1 0 for 0 (5) Hence, ....

[Article contains additional citation context not shown here]

I. Parberry, A primer on the complexity theory of neural networks, in: R.B. Banerji, ed., Formal Techniques in Artificial Intelligence: A Sourcebook, Vol. 6: Studies in Computer Science and Artificial Intelligence, Elsevier, North--Holland, Amsterdam (1990) 217--268.

....of conventional computers are by now well understood in terms of classical models of computation such as Turing machines. Analogously, many fundamental results have been achieved in the past decade concerning the capabilities of neural networks for general computation [26] 45] 86] 90] 94] [95], 96] 105] 110] 128] 136] In particular, the computational and descriptive powers of neural nets have been investigated by comparing their various architectures with each other and with more traditional computational models and descriptive tools such as nite automata, regular ....

.... employ the hard limiter or threshold activation function ( 1 for 0 0 for 0 : 3) Sometimes when more appropriate, bipolar values f1; 1g (or even more general discrete domains) can be substituted for binary ones f0; 1g without any substantial change in the size of weights [95], 96] Analog state networks, on the other hand, approximate (3) with some continuous sigmoid activation function, e.g. the saturated linear function ( 8 : 1 for 1 for 0 1 0 for 0 (4) or the standard (logistic) sigmoid ( 1 1 e : 5) Hence the states of ....

[Article contains additional citation context not shown here]

I. Parberry, A primer on the complexity theory of neural networks, in: R.B. Banerji, ed., Formal Techniques in Articial Intelligence: A Sourcebook (Vol. 6: Studies in Computer Science and Articial Intelligence, Elsevier, North{Holland, Amsterdam, 1990), 217-268.

....criticism of Minsky and Papert [65] The intent of this paper is to survey some of the central results in the complexity theory of neural network computation, as developed to date. We give no proofs. The paper might be most profitably read in conjunction with Ian Parberry s earlier survey article [72]. which also gives the proofs of many of the most significant (semi )elementary results. Our emphasis is at some points slightly different from Parberry s, and we also update his article with some of the more recent developments. Other survey articles of related topics, partially overlapping the ....

....and the sum total of the absolute values of all the weights in the network. Because real number parameters may be scaled at will, the weight measure really makes sense only for networks with integer parameters. The following result is fundamental ( 67, 66] for recent versions of the proof, see [37, 42, 72, 73, 80]) Theorem 3.1 Any threshold function on n variables can be computed by a threshold gate with integer weights w i such that jw i j (n 1) n 1) 2 =2 n , for all i = 0; n. A converse to this was proved only very recently [37] although weaker versions of the converse have been known ....

[Article contains additional citation context not shown here]

Parberry, I. A primer on the complexity theory of neural networks. In: Formal Techniques in Artificial Intelligence: A Sourcebook (ed. R. B. Banerji). Elsevier -- North-Holland, Amsterdam, 1990. Pp. 217--268.

.... models of computation such as Turing machines which are useful for understanding the computational potential and limits of conventional computers, the capability of neural networks to realize general computations have been studied for more than decade and many relevant results have been achieved [25, 44, 84, 88, 92, 93, 94, 103, 108, 123, 131]. In particular, the computational and descriptive power of neural nets have been investigated by comparing their various architectures with each other and with more traditional computational models and descriptive tools including nite automata, regular expressions, grammars, Turing machines, ....

....employ the hard limiter or threshold activation function ( 1 for 0 0 for 0 : 2. 3) Sometimes when more appropriate, bipolar values f1; 1g (or even more general discrete domains) can be substituted for binary ones f0; 1g without any substantial change in the size of weights [93, 94]. The analog state networks, on the other hand, approximate (2.3) with some continuous sigmoid activation function, e.g. the saturated linear function ( 8 : 1 for 1 for 0 1 0 for 0 (2.4) or the standard (logistic) sigmoid ( 1 1 e : 2.5) Hence the states ....

[Article contains additional citation context not shown here]

I. Parberry, A primer on the complexity theory of neural networks, in: R.B. Banerji, ed., Formal Techniques in Articial Intelligence: A Sourcebook (Vol. 6: Studies in Computer Science and Articial Intelligence, Elsevier, North{ Holland, Amsterdam, 1990), 217-268.

....the criticism of Minsky and Papert [51] The intent of this paper is to survey some of the central results in the complexity theory of neural network computation, as developed to date. We give no proofs. The paper might be most profitably read in conjunction with Ian Parberry s earlier survey [57], which also gives the proofs of many of the most significant (semi ) elementary results. Our emphasis is at some points slightly different from Parberry s, and we also update his survey with some of the more recent developments. Other surveys of related topics, partially overlapping the present ....

....and the sum total of the absolute values of all the weights in the network. Because real number parameters may be scaled at will, the weight measure really makes sense only for networks with integer parameters. The following result is fundamental ( 53, 52] for recent versions of the proof, see [31, 35, 57, 58, 62]) Theorem 1. Any linearly separable Boolean function on n variables can be implemented by a perceptron with integer weights w i such that jw i j (n 1) n 1) 2 =2 n , for all i = 0; n. A converse to this was proved only very recently [31] although weaker versions of the converse ....

[Article contains additional citation context not shown here]

Parberry, I. A primer on the complexity theory of neural networks. In: Formal Techniques in Artificial Intelligence: A Sourcebook (ed. R. B. Banerji). Elsevier -- NorthHolland, Amsterdam, 1990. Pp. 217--268.

....networks [29, 30] In this overview, we focus on the computational power of recurrent (cyclic) neural networks. The computational study of feedforward (acyclic) networks has intimate connections to the classical theory of Boolean circuit complexity [53] and is surveyed brie y in the articles [36, 40], and at greater depth in the books [41, 43, 52] 2. BASIC NOTIONS AND RESULTS With the brief exception of Section 6 on continuous time models, we shall be concerned with nite discrete time recurrent networks. Such a network consists of n computational units or neurons, indexed as 1; n, ....

Parberry, I. A primer on the complexity theory of neural networks. In Formal Techniques in Articial Intelligence: A Sourcebook (ed. R. B. Banerji). Elsevier { North-Holland, Amsterdam, 1990. Pp. 217{ 268.

....computation time of a cyclic network is not bounded, then it is fairly easy to see that the class of functions computed by polynomial size asymmetric nets equals the class PSPACE poly of functions computed by polynomial space bounded Turing machines with polynomially bounded advice. Parberry in [27] attributes this result to an early unpublished report [23] but for completeness we outline a proof in Section 3. Unbounded computation time might not be a totally unreasonable modeling assumption when one considers systems with a potential for an extremely fast analog or optical ....

....nets are computationally equivalent to asymmetric ones, i.e. capable of computing all of P poly. For a general introduction to neural computation, see the excellent textbook [16] for aspects of recurrent networks, see [19] and for computational complexity issues, see the survey papers [26, 27, 32]. 2 Preliminaries As in [27] we define a (discrete) neural network as a 6 tuple N = V; I; O; A; w; h) where V is a finite set of units, which we assume are indexed as V = f1; pg; I V and O V are sets of input and output units, respectively; A V is a set of initially active ....

[Article contains additional citation context not shown here]

Parberry, I. A primer on the complexity theory of neural networks. In: Formal Techniques in Artificial Intelligence: A Sourcebook (ed. R. B. Banerji). Elsevier -- NorthHolland, Amsterdam, 1990. Pp. 217--268.

....in Hopfield nets is the MIN ENERGY or GROUND STATE problem of finding a network state with minimal energy (2.1) for a given symmetric neural network. Remember that in (2.1) it is assumed for the simplicity that w jj = 0 and w j0 = 0 for j = 1; n. In addition, without loss of generality [21], throughout this section, we will work with frequently used bipolar states Gamma1; 1 of neurons instead of binary ones 0; 1 introduced in (1.2) where 0 is now replaced by Gamma1. This problem appears to be of a special interest since many hard combinatorial optimization problems were ....

Parberry, I. A primer on the complexity theory of neural networks. In Formal Techniques in Artificial Intelligence: A Sourcebook, editor R. B. Banerji, 217--268, Elsevier, North-Holland, Amsterdam, 1990.

....threshold circuits of size S(n) TC 0 will denote TC 0 (n O(1) TC 0 captures the complexity of important natural computational problems such as sorting, counting, and integer multiplication. It is also a good complexity theoretic model for the neural net model of computation [Par90] It is easy to observe that ACC 0 TC 0 (for example, see [BIS90] and thus we have even fewer lower bounds for the threshold circuit model than for ACC 0 circuits. Furthermore, since TC 0 (s(n) DSPACE(log s(n) for s(n) n) and since (by an easy consequence of the space hierarchy ....

I. Parberry. A primer on the complexity theory of neural networks. In R. Banerji, editor, Formal Techniques in Artificial Intelligence: A Sourcebook, volume 6 of Studies in Computer Science and Artificial Intelligence, pages 217--268. North-Holland, Amsterdam, 1990.

.... Threshold circuits are studied in part because MAJORITY gates have roughly the same computational power as integer multiplication gates (Chandra et al. 1984) and also because the neural net model of the brain is computationally equivalent to a threshold circuit (Parberry and Schnitger (1989) Parberry (1990)) Little is known about depth reduction in the context of threshold circuits; the best results in this direction are the results of Hajnal et al. 1987) where it was shown that there is a language recognized by a family of polynomialsize depth three majority circuits that cannot be recognized by ....

Parberry, I. (1990), A primer on the complexity theory of neural networks, in "Formal Techniques in Artificial Intelligence: A Sourcebook" (R. Banerji, Ed.), Studies in Computer Science and Artificial Intelligence 6, pp. 217--268, North-Holland, Amsterdam.

....threshold function and t is known as the threshold. It is a wellknown fact that the possibly infinite information contained in the real components of the weight vector can be made finite without restricting the class of representable functions by requiring all weights to be integers (see e.g. [6, 8, 12] for proofs) In the past, there has been considerable interest to bound the maximum absolute integral value sufficient for the weights from above for various reasons. Predominant was the search for a polynomial upper bound on the length of a weight in binary representation leading to O(n log n) ....

....for proofs) In the past, there has been considerable interest to bound the maximum absolute integral value sufficient for the weights from above for various reasons. Predominant was the search for a polynomial upper bound on the length of a weight in binary representation leading to O(n log n) [5, 11, 12, 13]. The tightest result has been given by Muroga [9] Investigating weight vectors satisfying the so called normalized system of inequalities w 1 x 1 Delta Delta Delta w n x n t if f(x) 1 w 1 x 1 Delta Delta Delta w n x n t Gamma 1 if f(x) 0 (2) he obtained the following ....

[Article contains additional citation context not shown here]

I. Parberry. A primer on the complexity theory of neural networks. In R. B. Banerji, editor, Formal Techniques in Artificial Intelligence: A Sourcebook, pages 217--268. Elsevier Science Publishers B. V. (North-Holland), Amsterdam, 1990.

....in polynomial time within a factor n 1 Gammaffl for any fixed ffl 0, unless P = NP . Similar results for other problems arising in the context of analyzing Hopfield nets have been obtained in [1, 3] For general introductions to computational complexity issues in neural networks, see [8, 10, 11]. We start by examining the easier case: the synchronous network. As will be seen, the boundary between tractability and intractability is here located between computing direct (one step) and two step attraction radii. We first observe that the former can be computed in polynomial time. Theorem ....

I. Parberry, "A primer on the complexity theory of neural networks," in Formal Techniques in Artificial Intelligence: A Sourcebook, ed. R. B. Banerji. Elsevier, Amsterdam, 1990, 217--268.

....computation time of a cyclic network is not bounded, then it is fairly easy to see that the class of functions computed by polynomial size asymmetric nets equals the class PSPACE poly of functions computed by polynomial space bounded Turing machines with polynomially bounded advice. Parberry in [29] attributes this result to an early unpublished report [25] but for completeness we outline a proof in Section 3. Unbounded computation time might not be a totally unreasonable modeling assumption when one considers systems with a potential for extremely fast analog or optical implementations. ....

....symmetric nets are computationally equivalent to asymmetric ones, i.e. capable of computing all of P poly. For a general introduction to neural computation, see the excellent textbook [17] for aspects of recurrent networks, see [21] and for computational complexity issues, see the survey papers [28, 29, 35]. 2 Preliminaries As in [29] we define a (discrete) neural network as a 6 tuple N = V; I; O; A; w; h) where V is a finite set of units, which we assume are indexed as V = f1; pg; I V and O V are the sets of input and output units, respectively; A V is a set of initially active ....

[Article contains additional citation context not shown here]

Parberry, I. A primer on the complexity theory of neural networks. In: Formal Techniques in Artificial Intelligence: A Sourcebook (ed. R. B. Banerji). Elsevier -- North-Holland, Amsterdam, 1990. Pp. 217--268.

.... networks are those by Kamp and Hasler [46] and by Hertz, Krogh and Palmer [38] Related network models for associative memory applications are discussed by Kohonen in [53] For general introductions to computational complexity issues in neural networks, see the book [70] and the survey articles [68, 69, 87]. The most typical applications of Hopfield networks are as associative memories, and for solving combinatorial optimization problems. An (auto)associative content addressable memory is a storage device that stores vectors u i , for i = 1, 2, m, in such a way that upon presentation of an ....

I. Parberry. A primer on the complexity theory of neural networks. In R. B. Banerji, ed. Formal Techniques in Artificial Intelligence: A Sourcebook, pp. 217--268. Elsevier, Amsterdam, 1990.

....threshold circuits of size S(n) TC 0 will denote TC 0 (n O(1) TC 0 captures the complexity of important natural computational problems such as sorting, counting, and integer multiplication. It is also a good complexity theoretic model for the neural net model of computation [Par90] It is easy to observe that ACC 0 TC 0 , and thus we have even fewer lower bounds for the threshold circuit model than for ACC 0 circuits. It is an easy consequence of the space hierarchy theorem that PSPACE complete sets require exponential size uniform TC 0 circuits, but there is ....

I. Parberry. A primer on the complexity theory of neural networks. In R. Banerji, editor, Formal Techniques in Artificial Intelligence: A Sourcebook, volume 6 of Studies in Computer Science and Artificial Intelligence, pages 217--268. NorthHolland, Amsterdam, 1990.

....criticism of Minsky and Papert [65] The intent of this paper is to survey some of the central results in the complexity theory of neural network computation, as developed to date. We give no proofs. The paper might be most profitably read in conjunction with Ian Parberry s earlier survey article [72]. which also gives the proofs of A preliminary version of this paper appears in Proc. of the 17th International Symp. on Mathematical Foundations of Computer Science (Prag, Aug. 1992) Lecture Notes in Computer Science 629, Springer Verlag, Berlin, 1992, pp. 50 61. Received December 1995. 2 ....

....and the sum total of the absolute values of all the weights in the network. Because real number parameters may be scaled at will, the weight measure really makes sense only for networks with integer parameters. The following result is fundamental ( 67, 66] for recent versions of the proof, see [37, 42, 72, 73, 80]) Theorem 1. Any threshold function on n variables can be computed by a threshold gate with integer weights w i such that jw i j (n 1) n 1) 2 =2 n , for all i = 0; n. A converse to this was proved only very recently [37] although weaker versions of the converse have been known ....

[Article contains additional citation context not shown here]

Parberry, I. A primer on the complexity theory of neural networks. In: Formal Techniques in Artificial Intelligence: A Sourcebook (ed. R. B. Banerji). Elsevier -- North-Holland, Amsterdam, 1990. Pp. 217--268.

....hand, if computation times are not bounded, then a relatively straightforward argument shows that the class of functions computed by polynomial size asymmetric nets equals the class PSPACE poly of functions computed by polynomial space bounded Turing machines with polynomially bounded advice. Parberry (1990) attributes this result to an early unpublished report by Lepley and Miller (1983) but for completeness we outline a proof in Section 3. Thus, general asymmetric recurrent nets are fairly easy to characterize computationally, and turn out to be quite powerful. On the other hand, as pointed out by ....

....time bounded one. For more information on the computational aspects of recurrent threshold logic networks, or more generally automata networks, see, e.g. the survey articles and books by Flor een and Orponen (1994) Fogelman et al. 1987) Goles and Mart inez (1990) Kamp and Hasler (1990) and Parberry (1990, 1994) 2 Preliminaries Following Parberry (1990) we define a (discrete) neural network as a 6 tuple N = V; I ; O;A; w; h) where V is a finite set of units, which we assume are indexed as V = f1; pg; I V and O V are the sets of input and output units, respectively; A V is a ....

[Article contains additional citation context not shown here]

Parberry, I. 1990. A primer on the complexity theory of neural networks. In: Formal Techniques in Artificial Intelligence: A Sourcebook (ed. R. B. Banerji). Elsevier -- North-Holland, Amsterdam. Pp. 217--268.

.... Threshold circuits are studied in part because MAJORITY gates have roughly the same computational power as integer multiplication gates (Chandra et al. 1984) and also because the neural net model of the brain is computationally equivalent to a threshold circuit (Parberry and Schnitger (1989) Parberry (1990)) Little is known about depth reduction in the context of threshold circuits; the best results in this direction are the results of Hajnal et al. 1987) where it was shown that there is a language recognized by a family of polynomialsize depth three majority circuits that cannot be recognized by ....

Parberry, I. (1990), A primer on the complexity theory of neural networks, in "Formal Techniques in Artificial Intelligence: A Sourcebook" (R. Banerji, Ed.), Studies in Computer Science and Artificial Intelligence 6, pp. 217--268, North-Holland, Amsterdam.

....technology) but may scale with the size of the problem being solved. ffl Node function: gates may compute weighted majority functions. For more details on the subject matter of this paper, and the application of other branches of computational complexity theory to neural networks, see Parberry [29, 30]. The remainder of this paper is divided into four sections. The first considers a simple feedforward neural network model. The second examines computation with a version of this model that has node functions limited to Boolean conjunction, disjunction, and complement. The third considers ....

....circuit has size S(n)2 p log S(n) which for any ffl 2 IR and large enough n is less than S(n) 1 ffl . 2 A weaker form of Corollary 4.7 is due to Chandra, Stockmeyer, and Vishkin [7] Our result is the obvious generalization, and tightens the sloppy analysis of Theorem 5.2. 8 of Parberry [29]. 15 And And And And Or Or x 1 y x 2 x 1 x 2 y g g g g g g 1 2 3 4 5 6 Figure 9: An alternating circuit computing y = x 1 Phi x 2 and its complement. x 1 x 2 x 1 x 2 x x x x x x x x Parity 3 3 4 4 n 1 n n 1 n Parity Parity x x Parity x x n 2 n 2 n 3 n 3 y ....

I. Parberry. A primer on the complexity theory of neural networks. In R. Banerji, editor, Formal Techniques in Artificial Intelligence: A Sourcebook, volume 6 of Studies in Computer Science and Artificial Intelligence, pages 217--268. North-Holland, 1990.

No context found.

I. Parberry. A primer on the complexity theory of neural networks. In Formal Techniques in Artificial Intelligence: A Sourcebook. Elsevier, 1990.

No context found.

I. Parberry. A primer on the complexity theory of neural networks. In Formal Techniques in Artificial Intelligence: A Sourcebook. Elsevier, 1990.

No context found.

I. Parberry. A primer on the complexity theory of neural networks. In Formal Techniques in Artificial Intelligence: A Sourcebook. Elsevier, 1990.

No context found.

I. Parberry, A primer on the complexity theory of neural networks, in A Sourcebook on Formal Techniques in Artificial Intelligence, ed. R. Banerji, North-Holland.

First 50 documents

Online articles have much greater impact   More about CiteSeer.IST   Add search form to your site   Submit documents   Feedback  

CiteSeer.IST - Copyright Penn State and NEC