#### DMCA

## Circuit Complexity and Feedforward Neural Networks (1996)

Venue: | in Mathematical Perspectives on Neural Networks |

Citations: | 3 - 0 self |

### Citations

10437 | Introduction to algorithms - Cormen, Leiserson, et al. - 2001 |

409 | The Complexity of Boolean Functions
- Wegener
- 1987
(Show Context)
Citation Context ...al size. For example, the sum of two n-bit natural numbers can be computed by an alternating circuit of size O(n 2 ) and depth 4 using the standard carrylookahead algorithm (see, for example, Wegener =-=[40]-=-). Chandra, Fortune and Lipton have shown by a sophisticated argument that the size bound can be reduced from O(n 2 ) to an almost linear function. Define f (1) (x) = f(x), and for i ? 1, f (i) (x) = ... |

233 |
A taxonomy of problems with fast parallel algorithms
- Cook
- 1985
(Show Context)
Citation Context ...C k to be the set of problems that can be solved in polynomial size, and depth O(log k n), for ks0. Clearly AC k ` AC k+1 for ks0, and AC = [ k0 AC k : The classes AC k for ks1 first appeared in Cook =-=[8]-=-. The relationship between classical circuits and AND-OR circuits is obvious. Theorem 4.3 For every finite alternating circuit of size s and depth d there is a finite classical circuit of size s(s + n... |

226 |
On the computational complexity of algorithms
- Hartmanis, Stearns
- 1965
(Show Context)
Citation Context ...g time, memory, hardware, and power. A theory of computation, called computational complexity theory 1 has grown from this simple observation, starting with the seminal paper of Hartmanis and Stearns =-=[14]-=-. The prime tenet of this field is that some computational problems intrinsically consume more resources than others. The resource usage of a computation is measured as a function of the size of the p... |

212 |
Threshold logic and its applications
- Muroga
- 1971
(Show Context)
Citation Context ...g is bounded above in magnitude by (n + 1) (n+1)=2 . Thus we deduce that j\Delta i js(n + 1) (n+1)=2 =2 n . 2 Theorem 5.1 is due to Muroga, Toda, and Takasu [23], and appears in more detail in Muroga =-=[22]-=-. Weaker versions of this result were more recently rediscovered by Hong [15], Raghavan [34], and Natarajan [25]. It is unknown whether the upper bound of Theorem 5.1 is tight. For obtaining lower bou... |

212 |
Separating the polynomial–time hierarchy by oracles
- Yao
- 1985
(Show Context)
Citation Context ...was originally proved by Furst, Saxe, and Sipser [11] and Ajtai [2], and improved lower bounds on the size required for constant depth alternating circuit for PARITY were successively obtained by Yao =-=[41]-=- and Hastad [38]. 2 More strongly, for every k 2 IN there are functions that can be computed by alternating circuits in polynomial size and depth k, but require exponential size alternating circuits o... |

192 |
A general framework for parallel distributed processing
- Rumelhart, Hinton, et al.
- 1986
(Show Context)
Citation Context ...etwork researchers is a finite network of simple computational devices wired together so that they interact and cooperate to perform a computation (see, for example, Rumelhart, Hinton, and McClelland =-=[36]-=-). Suppose F is a set of functions f : IB ! IB. A feedforward neural network with node function set F is a 5-tuple C = (V; X; Y; E; `), where V is a finite ordered set of gates X is a finite ordered s... |

140 |
Threshold circuits of bounded depth
- Hajnal, Maass, et al.
- 1993
(Show Context)
Citation Context ...hat is, T C 0 without Boolean negations, Yao [42]). In the general (nonmonotone) case, it is trivial to separate T C 0 depth 1 from depth 2, and depth 2 has been separated from depth 3 (Hajnal et al. =-=[12]-=-), but beyond that nothing is known. It is known that the sum of n polynomial-bit numbers can be computed by a threshold circuit in constant depth and polynomial size (Chandra, Stockmeyer and Vishkin[... |

131 |
Σ11-formulae on finite structures
- Ajtai
- 1983
(Show Context)
Citation Context ...es in NC1 that are not in AC0: Theorem 4.8 Every constant depth AND-OR circuit for PARITY requires exponential size. Proof: This result was originally proved by Furst, Saxe, and Sipser [11] and Ajtai =-=[2]-=-, and improved lower bounds on the size required for constant depth alternating circuit for PARITY were successively obtained by Yao [41] and Hastad [38]. More strongly, for every k ∈ IN there are f... |

127 |
Machine learning : a theoretical approach
- Natarajan
- 1991
(Show Context)
Citation Context ...eorem 5.1 is due to Muroga, Toda, and Takasu [23], and appears in more detail in Muroga [22]. Weaker versions of this result were more recently rediscovered by Hong [15], Raghavan [34], and Natarajan =-=[25]-=-. It is unknown whether the upper bound of Theorem 5.1 is tight. For obtaining lower bounds, it is useful to count the number of n-input weighted threshold functions. Theorem 5.2 There are at least 2 ... |

125 |
Constant depth reducibility
- Chandra, Stockmeyer, et al.
- 1984
(Show Context)
Citation Context ...)c: The resulting circuit has size S(n)2 p log S(n) , which for any ffl 2 IR + and large enough n is less than S(n) 1+ffl . 2 A weaker form of Corollary 4.7 is due to Chandra, Stockmeyer, and Vishkin =-=[7]-=-. Our result is the obvious generalization, and tightens the sloppy analysis of Theorem 5.2.8 of Parberry [29]. 15 And And And And Or Or x 1 y x 2 x 1 x 2 y g g g g g g 1 2 3 4 5 6 Figure 9: An altern... |

123 |
Parity, Circuits and the Polynomial Time Hierarchy
- Furst, Saxe, et al.
- 1984
(Show Context)
Citation Context ...ery small increase in size. AC 0 , the set of problems that can be solved in polynomial size and constant depth, is of particular interest. The class AC 0 was first studied by Furst, Saxe, and Sipser =-=[11]-=-, and was named by Barrington [4]. We saw in Theorem 3.3 that every Boolean function can be computed in constant depth with exponential size, but this cannot be considered practical for any but the ve... |

110 |
Two theorems on random polynomial time
- Adleman
- 1978
(Show Context)
Citation Context ...n=2 2 IB by simply taking the AND gate that computes g(b 1 ; : : : ; b n=2 )(x n=2 ; : : : ; x n ); and giving it extra inputs from x 1 [b 1 ]; : : : ; x n=2 [b n=2 ] (where x i [0] denotes x i , x i =-=[1]-=- denotes x i , x i [0] denotes x i , and x i [1] denotes x i ). The resulting circuit still has depth 2 and size 2 n=2+1 . Finally, we note that f(x 1 ; : : : ; x n ) = h(0; : : : ; 0 --- --zsn=2 )(x ... |

88 |
A note on the power of threshold circuits
- Allender
- 1989
(Show Context)
Citation Context ... AC 0 is contained in T C 0 depth 3 (Immerman and Landau [16]); however, all that is known is that every function in AC 0 can be computed by threshold circuits of depth 3 and size n log c n (Allender =-=[3]-=-). There is no depth hierarchy theorem for T C 0 , although there is a depth hierarchy theorem for monotone T C 0 (that is, T C 0 without Boolean negations, Yao [42]). In the general (nonmonotone) cas... |

86 |
1 -Formulae on Finite Structures
- Ajtái
- 1983
(Show Context)
Citation Context ...in NC 1 that are not in AC 0 : Theorem 4.8 Every constant depth AND-OR circuit for PARITY requires exponential size. Proof: This result was originally proved by Furst, Saxe, and Sipser [11] and Ajtai =-=[2]-=-, and improved lower bounds on the size required for constant depth alternating circuit for PARITY were successively obtained by Yao [41] and Hastad [38]. 2 More strongly, for every k 2 IN there are f... |

77 |
On simultaneous resource bounds
- PIPPENGER
- 1979
(Show Context)
Citation Context ...fine NC = [ k0 NC k : NC is an abbreviation of "Nick's Class", named by Cook [9] after Pippenger, who discovered an important relationship between NC and conventional Turing machine based co=-=mputation [33]-=-. Corollary 4.4 1. For ks0, NC k ` AC k . 2. For ks0, AC k ` NC k+1 . 3. NC = AC. 14 AND AND AND AND AND AND AND AND Figure 8: Replacing a fan-in 8 AND-gate with a tree of fan-in 2 gates. Proof: Part ... |

74 |
Borel sets and circuit complexity
- Sipser
- 1983
(Show Context)
Citation Context ...rongly, for every k 2 IN there are functions that can be computed by alternating circuits in polynomial size and depth k, but require exponential size alternating circuits of depth k \Gamma 1 (Sipser =-=[37]-=-). This is often called the depth hierarchy theorem for AC 0 . 17 5 Threshold Circuits The node function set for a weighted threshold circuit is the set of weighted majority functions f : IB n ! IB. T... |

71 |
The circuit value problem is log space complete for
- Ladner
- 1975
(Show Context)
Citation Context ...iption of A with a copy of the input x 1 ; : : : ; x n . The output is an instance of CVP which is a member of CVP iff x 2 A. Therefore, Asl CVP. 2 The uniform version of Theorem 4.2 is due to Ladner =-=[17]-=-. The proof in that reference is somewhat sketchy; a more detailed proof appears in Parberry [28]. P-complete problems are 13 P P-complete Problems CVP PARITY AC Figure 7: P-complete problems (conject... |

62 | On the computational power of sigmoid versus boolean threshold circuits
- Maass, Schnitger, et al.
- 1991
(Show Context)
Citation Context ...el is essentially the same as a discrete one within T C 0 (Obradovic and Parberry [26, 27]). More importantly, the same is true even without the assumption of robustness (Maass, Schnitger, and Sontag =-=[20]-=-). More specifically, any problem that can be solved by an analog neural network of polynomial size and constant depth can be approximated by a T C 0 circuit. 22 7 Conclusion Circuit complexity is a u... |

62 |
Circuit complexity and neural networks
- Parberry
- 1994
(Show Context)
Citation Context ...ay compute weighted majority functions. For more details on the subject matter of this paper, and the application of other branches of computational complexity theory to neural networks, see Parberry =-=[29, 30]-=-. The remainder of this paper is divided into four sections. The first considers a simple feedforward neural network model. The second examines computation with a version of this model that has node f... |

51 |
Unbounded Fan-in Circuits and Associative Functions
- Chandra, Fortune, et al.
- 1983
(Show Context)
Citation Context ... to an almost linear function. Define f (1) (x) = f(x), and for i ? 1, f (i) (x) = f(f (i\Gamma1) (x)). Define f 1 (n) = 2 n , and for i ? 1, f i (n) = f (n) i\Gamma1 (2). Chandra, Fortune and Lipton =-=[6]-=- have shown that there is an alternating circuit for computing the carry of two n-bit numbers in depth 6d+3 and size nf \Gamma1 d (n) 2 . Surprisingly, they also found a matching lower bound [5]. Howe... |

45 |
Parallel computation with threshold functions
- Parberry, Schnitger
- 1988
(Show Context)
Citation Context ...t numbers can be computed by a threshold circuit in constant depth and polynomial size (Chandra, Stockmeyer and Vishkin[7]). It can be deduced from this result and Theorem 5.1 (Parberry and Schnitger =-=[31]-=-) that any problem that can be solved using a polynomial size, constant depth weighted threshold circuit can be solved using a polynomial size, constant depth threshold circuit. That is, T C 0 is the ... |

42 | The complexity of iterated multiplication
- Immerman, Landau
- 1995
(Show Context)
Citation Context ...value kS f k + 1 whose inputs come from these gates. 2 The exact relationship between AC 0 and T C 0 is not known. It has been conjectured that AC 0 is contained in T C 0 depth 3 (Immerman and Landau =-=[16]-=-); however, all that is known is that every function in AC 0 can be computed by threshold circuits of depth 3 and size n log c n (Allender [3]). There is no depth hierarchy theorem for T C 0 , althoug... |

41 |
Parallel Complexity Theory
- Parberry
- 1987
(Show Context)
Citation Context ...s a member of CVP iff x 2 A. Therefore, Asl CVP. 2 The uniform version of Theorem 4.2 is due to Ladner [17]. The proof in that reference is somewhat sketchy; a more detailed proof appears in Parberry =-=[28]-=-. P-complete problems are 13 P P-complete Problems CVP PARITY AC Figure 7: P-complete problems (conjectured). interesting since, by Lemma 4.1, if one of them is in AC, then AC = P . If the conjecture ... |

35 |
A Towards a complexity theory of synchronous parallel computation. Extratt de L'Enstgetgntement Mathematique T XXVII, FASC
- COOK
- 1981
(Show Context)
Citation Context ...hat can be solved by classical circuits in polynomial size and depth O(log k n), for ks1. Clearly NC k ` NC k+1 for ks1. Define NC = [ k0 NC k : NC is an abbreviation of "Nick's Class", name=-=d by Cook [9]-=- after Pippenger, who discovered an important relationship between NC and conventional Turing machine based computation [33]. Corollary 4.4 1. For ks0, NC k ` AC k . 2. For ks0, AC k ` NC k+1 . 3. NC ... |

35 |
Linear function neurons: structure and training
- Hampson, Volper
- 1986
(Show Context)
Citation Context ...ed threshold functions have weights strictly less than this value, then there would be less than 2 n(n\Gamma1)=2 weighted threshold functions. A similar lower bound can be found in Hampson and Volper =-=[13]-=-. This nonconstructive counting argument is a little unsatisfactory, since it does not give any specific weighted threshold functions with weights this large. Fortunately, a specific example is known:... |

34 | On connectionist models
- Hong
- 1988
(Show Context)
Citation Context ...lta i js(n + 1) (n+1)=2 =2 n . 2 Theorem 5.1 is due to Muroga, Toda, and Takasu [23], and appears in more detail in Muroga [22]. Weaker versions of this result were more recently rediscovered by Hong =-=[15]-=-, Raghavan [34], and Natarajan [25]. It is unknown whether the upper bound of Theorem 5.1 is tight. For obtaining lower bounds, it is useful to count the number of n-input weighted threshold functions... |

26 |
Theory of majority decision elements
- Muroga, Todo, et al.
(Show Context)
Citation Context ...+ 1) \Theta (n + 1) matrix over f\Gamma1; 1g is bounded above in magnitude by (n + 1) (n+1)=2 . Thus we deduce that j\Delta i js(n + 1) (n+1)=2 =2 n . 2 Theorem 5.1 is due to Muroga, Toda, and Takasu =-=[23]-=-, and appears in more detail in Muroga [22]. Weaker versions of this result were more recently rediscovered by Hong [15], Raghavan [34], and Natarajan [25]. It is unknown whether the upper bound of Th... |

26 |
A primer on the complexity theory of neural networks
- Parberry
- 1990
(Show Context)
Citation Context ...ay compute weighted majority functions. For more details on the subject matter of this paper, and the application of other branches of computational complexity theory to neural networks, see Parberry =-=[29, 30]-=-. The remainder of this paper is divided into four sections. The first considers a simple feedforward neural network model. The second examines computation with a version of this model that has node f... |

23 |
Lower Bounds for Constant Depth Circuit for
- Chandra, Fortune, et al.
- 1983
(Show Context)
Citation Context ...ipton [6] have shown that there is an alternating circuit for computing the carry of two n-bit numbers in depth 6d+3 and size nf \Gamma1 d (n) 2 . Surprisingly, they also found a matching lower bound =-=[5]-=-. However, there are languages in NC 1 that are not in AC 0 : Theorem 4.8 Every constant depth AND-OR circuit for PARITY requires exponential size. Proof: This result was originally proved by Furst, S... |

21 |
Circuits and local computation
- Yao
- 1989
(Show Context)
Citation Context ...epth 3 and size n log c n (Allender [3]). There is no depth hierarchy theorem for T C 0 , although there is a depth hierarchy theorem for monotone T C 0 (that is, T C 0 without Boolean negations, Yao =-=[42]-=-). In the general (nonmonotone) case, it is trivial to separate T C 0 depth 1 from depth 2, and depth 2 has been separated from depth 3 (Hajnal et al. [12]), but beyond that nothing is known. It is kn... |

20 |
Enumeration of Threshold Functions of Eight Variables
- Muroga, Tsuboi, et al.
- 1970
(Show Context)
Citation Context ...i + 1) ? 2 n(n\Gamma1)=2 : 2 This result is attributed to Dahlin by Muroga [22]. The lower-bound can be improved to C(n) ? 2 n(n\Gamma1)=2 +16 by observing that C(8) ? 2 44 (Muroga, Tsuboi, and Baugh =-=[24]-=-). Define the weight of a weighted threshold function f to be the smallest w 2 IN such that f has an integer presentation with all weights no greater than w in absolute value. We can deduce from Theor... |

16 |
Learning in threshold networks
- Raghavan
- 1988
(Show Context)
Citation Context ... (n+1)=2 =2 n . 2 Theorem 5.1 is due to Muroga, Toda, and Takasu [23], and appears in more detail in Muroga [22]. Weaker versions of this result were more recently rediscovered by Hong [15], Raghavan =-=[34]-=-, and Natarajan [25]. It is unknown whether the upper bound of Theorem 5.1 is tight. For obtaining lower bounds, it is useful to count the number of n-input weighted threshold functions. Theorem 5.2 T... |

15 |
Relating Boltzmann machines to conventional models of computation
- Parberry, Schnitger
- 1989
(Show Context)
Citation Context ...o far. Variations that appear in the literature include: 1. Networks with cycles. These can be unwound into feedforward neural networks in the obvious manner (see, for example, Parberry and Schnitger =-=[31, 32]-=-). 2. Probabilistic neural networks. Allowing computers access to a random source appears to make them more efficient than a plain deterministic computer in some circumstances (see, for example, Corme... |

11 |
Bounded width polynomial size branching programs recognize exactly those ‘languages
- BARRINGTON
- 1986
(Show Context)
Citation Context ... the set of problems that can be solved in polynomial size and constant depth, is of particular interest. The class AC 0 was first studied by Furst, Saxe, and Sipser [11], and was named by Barrington =-=[4]-=-. We saw in Theorem 3.3 that every Boolean function can be computed in constant depth with exponential size, but this cannot be considered practical for any but the very smallest values of n. Unfortun... |

11 |
Improved lower bounds for small depth circuits
- Hastad
- 1986
(Show Context)
Citation Context ...roved by Furst, Saxe, and Sipser [11] and Ajtai [2], and improved lower bounds on the size required for constant depth alternating circuit for PARITY were successively obtained by Yao [41] and Hastad =-=[38]-=-. 2 More strongly, for every k 2 IN there are functions that can be computed by alternating circuits in polynomial size and depth k, but require exponential size alternating circuits of depth k \Gamma... |

10 |
Learning with discrete multivalued neurons
- Obradović, Parberry
- 1994
(Show Context)
Citation Context ...alue). It can be shown that if one assumes that neuron outputs are robust to small errors in precision, then their model is essentially the same as a discrete one within T C 0 (Obradovic and Parberry =-=[26, 27]-=-). More importantly, the same is true even without the assumption of robustness (Maass, Schnitger, and Sontag [20]). More specifically, any problem that can be solved by an analog neural network of po... |

9 |
Implementing the algebra of logic functions in terms of bounded depth formulas in the basis f; ; :g: Soviet Phys
- Lupanov
- 1961
(Show Context)
Citation Context ...tance: x 1 ; : : : ; x n 2 IB. Question: Is kfi j x i = 1gk odd? Theorem 3.4 Any depth 2 alternating circuit for computing PARITY must have size at least 2 n\Gamma1 + 1. Theorem 3.4 is due to Lupanov =-=[18, 19]-=-. An obvious question to ask is whether we can reduce the size and trade it for increased depth. The answer is that this is not possible beyond a certain size: some functions intrinsically require exp... |

6 |
Analog neural networks of limited precision I: Computing with multilinear threshold functions (Preliminary version
- Obradovic, Parberry
- 1990
(Show Context)
Citation Context ...alue). It can be shown that if one assumes that neuron outputs are robust to small errors in precision, then their model is essentially the same as a discrete one within T C 0 (Obradovic and Parberry =-=[26, 27]-=-). More importantly, the same is true even without the assumption of robustness (Maass, Schnitger, and Sontag [20]). More specifically, any problem that can be solved by an analog neural network of po... |

2 |
Synthesis of threshold element networks for certain classes of Boolean functions
- Redkin
- 1970
(Show Context)
Citation Context ...ch is smaller by a polynomial amount (actually, a square-root) than the upper bound of Theorem 3.3, be met? Surprisingly it can be met with a circuit of depth 3, a result that can be traced to Redkin =-=[35]-=-. 8 Theorem 3.6 If f : IB n ! IB, then there is an alternating circuit of size O(2 n=2 ) and depth 3 that computes f . Proof: Let f : IB n ! IB. Without loss of generality, assume n is even (a similar... |

1 |
On the size of weights for threshold gates. Unpublished Manuscript
- Hastad
- 1992
(Show Context)
Citation Context ... Therefore, by induction on i, w i+1s2 i for 1sisk. In particular w k+1s2 k . 2 The above result can easily be modified to give an improved lower bound of bOE n = p 5c, where OE = (1 + p 5)=2. Hastad =-=[39]-=- has found a weighted threshold function that requires weights at least n n=2\GammaO(n) (see Parberry [30] for a slightly better bound on this function). Theorem 5.1 implies that the weights used in a... |