Karpinski, M., Macintyre, A. J. 1997. Polynomial bounds for VC dimension of sigmoidal and general Pfa#an neural networks, Journal of Computer and System Sciences, 54:169--176.  Home/Search   Document Not in Database   Summary   Related Articles   Check

.... But it is possible to obtain another bound using a different technique that has its roots in the proof of the k = 1 case given by Cover [6] and which can be traced back to Schlafli [16] Generalizations of this technique have recently proven useful for more complex partitioning methods; see [11, 4, 12, 3]. by k parallel hyperplanes is bounded as follows: Proof: Let N points, x 1 ; x 2 ; xN 2 R be given. Now, each partition by k parallel hyperplanes can be described by a permissible parameter vector p = w 1 ; w 2 ; w n ; 1 ; 2 ; k ) where, to say ....

M. Karpinski and A. MacIntyre. Polynomial bounds for VC dimension of sigmoidal and general Pfaffian neural networks. Journal of Computer and System Sciences, 54, 1997: 169--176.

....of the VC dimension. There has been considerable study of the VC dimension for various neural network systems. See, for example, Koiran and Sontag (1997, lower bounds, for networks with continuous activations) Baum and Haussler (1989, upper and lower bounds, for hard threshold networks) and Karpinski and Macintyre (1997, upper bounds, for sigmoidal and Pfaffian networks) Mixtures of Experts (ME) Jacobs et al. 1991) and Hierarchical Mixtures ofExperts (HME) Jordan and Jacobs 1994) are different types of neural network systems, which generalize the usual mixture models in the statistics literature ....

....6 in Section 2, applying Theorem 1, we have m = VCdim(H m B ( VCdim(H m B ( s ) VCdim(H m L ( s ) proving the theorem. 2 11 Proof of Theorem 3: The inequality is proved in the previous theorem. To prove the upper bound for VCdim(H m L ( s ) we apply the results of Karpinski and Macintyre (1997). In our situation, any classifier in H m L ( s ) has a decision boundary p(x) Gamma c = 0, or equivalently, m X j=1 e v j u T j x )f m Y k=1 (1 e Gammaff k Gammafi T k x )gfp(x) Gamma cg = 0: Note that is a polynomial on the q = 2m Gamma 1 elements e j , j ....

[Article contains additional citation context not shown here]

Karpinski, M. and Macintyre, A. 1997. Polynomial bounds for VC dimension of sigmoidal and general Pfaffian neural networks. Journal of Computer and System Science. 54, 169-176.

....which in turn relies on the results of Goldberg and Jerrum [6] on shatter coefficients of polynomial networks. A statement applicable to the general case X # IR d will also be made. The proof in this case relies on some recent developments of geometric techniques by Karpinksi and Macintyre [9], making use of the number of connected components of zero sets of certain systems equations. For lack of space, this latter result will be stated without proof. Lemma 3.1 Let X i and x belong to the finite set D, D 1, D d , and set K(x) exp( #x# 2 h) Then the shatter coefficient ....

....10.1 in [4] This result, together with the exponential bound in (5) yield the appropriate result. The proof is very similar to the one given in [4] for the standard kernel case. In the case where x # IR d a more elaborate approach is required, based on the work of Karpinksi and McIntyre [9]. In this case we obtain, making use of some recent contributions from [1] an upper bound of the form log S( Cm) O(d 4 m 2 d log l) which is much cruder than the upper bound given in Lemma 3.1. This issue will be addressed in future work. 4 Practical Results Although the holdout method ....

M. Karpinksi and A. Macintyre. Polynomial Bounds for VC Dimension of Sigmoidal and General Pfaffian Neural Networks. Journal of Computer and System Science, 54:169--176, 1997.

....probability measure on # , and let ## # (0# 1) be arbitrary constants. Then, #(## ## #) # if # # max # 16 # # log 4 # 32# # # log 32# # # # The next theorem gives an upper bound for the VC dimension of the class # ####### and is due essentially to Karpinski and Macintyre [38]. We cite it from [66] Theorem 3: The following upper bound holds: ######(# ####### ) # 2#log(4###)# Following Vidyasagar [67] 68] our initial purpose is to explore the utility of statistical learning theory in the ecient design of robust controllers for linear uncertain systems. Throughout ....

M. Karpinski and A.J. Macintyre. Polynomial bounds for VC dimension of sigmoidal and general Pfaan neural networks. J. Comp. and Sys. Sci., 54(1):169-176, February 1997. ##

.... conditions, FC cannot shatter any set of cardinality larger than p(smn) Conversely, if FC has VC dimension bounded by q(smn) then C has at most (e2 m =q(smn) q(smn) satis able sign conditions by Sauer s lemma (see for instance [5] and the references there) 2 For instance, it follows from [11] that if we expand the ordered eld of the real numbers with the exponential function, or even with Pfaan functions, the resulting structure still has few sign conditions. It is however not known whether these structures have ecient enumeration of sign conditions. If one wishes to obtain sharp ....

M. Karpinski and A. Macintyre. Polynomial bounds for VC dimension of sigmoidal and general Pfaan neural networks. Journal of Computer and System Sciences, 54:169-176, 1997.

....neural networks. On the one hand, we use constructions provided by Koiran and Sontag [10] to build networks that have large Vapnik Chervonenkis dimension and consist of gates that compute certain arithmetic functions. On the other hand, we follow the lines of reasoning of Karpinski and Macintyre [7] to derive an upper bound for the VapnikChervonenkis dimension of these networks from the estimates of Khovanskii [8] and a result due to Warren [16] In the following section we give the definitions of sigmoidal networks and the VapnikChervonenkis dimension. Then we present the lower bound ....

....Then we present the lower bound result for function approximation. Finally, we conclude with some discussion and open questions. 2 Sigmoidal Neural Networks and VC Dimension We briefly recall the definitions of a sigmoidal neural network and the VapnikChervonenkis dimension (see, e.g. [7, 10]) We consider feedforward neural networks which have a certain number of input nodes and one output node. The nodes which are not input nodes are called computation nodes and associated with each of them is a real number t, the threshold. Further, each edge is labelled with a real number w called ....

[Article contains additional citation context not shown here]

M. Karpinski and A. Macintyre. Polynomial bounds for VC dimension of sigmoidal and general Pfaffian neural networks. Journal of Computer and System Sciences, 54:169--176, 1997.

...., the function q f : U 0 = U Theta R f0; 1g : u; y) 7 H(f(u) Gamma y) as well as the class F 0 consisting of all such q f . The pseudo dimension of F is defined by: pd (F) vc (F 0 ) This definition is equivalent to the one in [7] We now review a basic pseudodimension estimate from [9], specialized to a form useful for our purposes. Let oe = tanh. Assume given a function : R l Theta R h R0 which can be expressed as: z) P (oe(R 1 ( z) oe(R n ( z) z) 6) where R 1 ( z) R n ( z) are polynomials each of (total) degree at most r Gamma ....

....and the degree and length of the Pfaffian chains involved. Precisely, we apply Theorem 10.7 with b instead of 2 log 2 B and Lemma 10.6 with q; d; D respectively equal to our n; q; r, to obtain respectively equations (7) and (8) The results cited from the book [20] are based on the paper [9]; the only minor difference is in the slight improvement in (7) which the original paper had given as b 16l. Some algebraic manipulation gives: Corollary 2.6 When r = 3, l 3n n 2 , n 5, and q 6k, pd (F) 3n 6 5n 3 log 2 k. 2 2.3 Uniform Approximation of Expected Loss The last ....

Karpinski, M., and A. Macintyre, "Polynomial bounds for VC dimension of sigmoidal and general Pfaffian neural networks," J. Computer Sys. Sci., to appear. (Summary in "Polynomial bounds for VC dimension of sigmoidal neural networks," in Proc. 27th ACM Symposium on Theory of Computing, 1995 , pp. 200-208.)

....network inputs x 2 R n and arbitrary assignments w 2 R m to the weights and biases of these units. We take as the class G of functions considered in the proof of Theorem 5.1 all functions of the form (x; q) z(F (x; w) q) for arbitrary parameters w 2 R m . The results presented in [Karpinski, Macintyre, 1995] imply that the pseudo dimension of this class G of functions is bounded by a polynomial in m, for all common choices of activation functions of the sigmoidal units and all practically relevant density kernels z for the noise process (even involving the exponential function) In the case where the ....

M. Karpinski, A. Macintyre, Polynomial bounds for VCdimension of sigmoidal and general Pfaffian neural networks. J. of Computer and System Sciences, to appear.

.... the application of these concepts to neural networks and a survey of recent work on that field the reader may be referred to Anthony [4] In the following we show that an upper bound for the VC dimension of the FA can be trivially derived by directly applying the result of Karpinski and Macintyre [38] (VC dimension of multilayer feedforward networks) and the extension of Koiran and Sontag [42] VC dimension of RNN) Theorem 2 (VC Dimension) For folding architectures with first order connections being constrained (by discretized interpretation of the output) to implement functions Xi 2 F of ....

Marek Karpinski and Angus Macintyre. Polynomial Bounds for VC Dimension of Sigmoidal and General Pfaffian Neural Networks. Journal of Computer and Systems Sciences, 54(1):169-- 176, February 1997.

....continuity restriction. Here we consider sigmoidal functions of two types: piecewise polynomial functions and the standard function of the form (x) 1 e Gammax ) Gamma1 . These results make use of known results from the analysis of multi variate algebraic polynomials ( 24] 25] [13], see review in [22] Before presenting the main results of this work, we should stress that the general conclusions presented here have broad applicability beyond the field of function approximation. It has become clear in recent years that efficient and robust strategies exist, whereby ....

M. Karpinski, A.J. Macintyre, Polynomial bounds for VC dimension of sigmoidal and general Pfaffian neural networks, Journal of Computer and System Science, 54, pp. 160-176, 1997.

....approximation error bounds, which have not in general been surmounted to date. Recently it has been shown that the VC dimension of many types of neural networks with continuous activation functions is finite even without imposing any conditions on the magnitudes of the parameters [MS93] GJ95][KM97]. Since there is a close connection between the VC dimension and the estimation error (see Section 4) this result is significant in the context of learning. Thus, as long as the function itself is bounded, one may proceed to derive good upper bounds for the covering numbers needed in establishing ....

....by (4) but with constants that depend on f . In the case of a single hidden layer network with k hidden units, the number of parameters W is given by W = k(d 2) 1 = O(k) Thus we observe that the final term in (4) is of order O(k log k log n=n) For the standard sigmoid, we have from [KM97] that Pdim = O(W 2 k 2 ) O(k 4 ) and thus this term is O(k 4 log n=n) Keeping in mind that the approximation error terms are identical up to a logarithmic factor log k, we can see that the guaranteed upper bound on the rates of convergence of the loss in the case of piecewise polynomial ....

M. Karpinksi and A. Macintyre. Polynomial Bounds for VC Dimension of Sigmoidal and General Pfaffian Neural Networks. Journal of Computer and System Science, 54:169--176, 1997.

.... For each A and k, by unfolding the iterations, one may also see the class BA;k as a class of classifiers representable by feedforward neural nets (with k hidden layers ) This trivial fact allows one to easily obtain estimates, based on those bounds which were developed (cf. 4, 1] 7] [10]) for the feedforward case. Theorem 1 For recurrent architectures, with w weights receiving inputs of length k: 1. The VC dimension of threshold recurrent architectures is O(kw log kw) 2. If oe : R R is a fixed piecewise polynomial function, the VC dimension of recurrent architectures with ....

.... 1 for d = 1 and D k 2d k 1 for d 2. 2 Proof of Theorem 1 (standard sigmoidal case) By unfolding, the recurrent architecture can be simulated by a feedforward net with kn nodes, where n be the number of nodes in the original architecture, and the same number w of programmable parameters. By [10] there is a O( kn) 2 w 2 ) upper bound on the VC dimension of that architecture. This is O(k 2 w 4 ) as claimed. Note: one can argue that the feedforward architecture has kw weights, but many of those weights are shared and there are only w n 2w programmable parameters. The result in ....

[Article contains additional citation context not shown here]

M. Karpinski and A. Macintyre, "Polynomial bounds for VC dimension of sigmoidal and general Pfaffian neural networks," J. Computer Sys. Sci. 54(1997), 169-176.

.... gates is known to have VC dimension at most O(w log w) where w is the number of weights (Baum and Haussler, 1989) Networks using piecewise polynomial functions for their gates have VC dimension O(w 2 ) Goldberg and Jerrum, 1995) whereas for sigmoidal networks the bound O(w 4 ) is known (Karpinski and Macintyre, 1997). With respect to lower bounds threshold networks with VC dimension Omega Gamma w log w) have been constructed (Sakurai, 1993; Maass, 1994) Furthermore, Koiran and Sontag, 1997) have shown that there are neural networks with VC dimension Omega Gamma w 2 ) Among these are networks that ....

....upper bound O(n 4 ) for nonoverlapping sigmoidal networks. We give a brief account. Proposition 7 The class of nonoverlapping sigmoidal networks on n inputs has VC dimension O(n 4 ) Proof. The VC dimension of a sigmoidal neural network with w weights is O(w 4 ) This has been shown by (Karpinski and Macintyre, 1997). By Sauer s Lemma (see, e.g. Anthony and Biggs, 1992) the number of dichotomies induced by a class of functions with VC dimension d 2 on a set of cardinality m 2 can be bounded from above by m d . Thus a nonoverlapping sigmoidal network on n inputs induces at most m O(n 4 ) dichotomies ....

Karpinski, M. and Macintyre, A. (1997). Polynomial bounds for VC dimension of sigmoidal and general pfaffian neural networks. Journal of Computer and System Sciences, 54:169--176.

....letting d be the P dimension, probability q(M; ffl) is upper bounded as follows (see [6] Theorem 7.1) q(M; ffl) 8( 16e ffl ln 16e ffl ) d e GammaM ffl 2 =32 : 4) Reportedly, the computation of a P dimension on the basis of its definition is a hard task. Quite recently, 7] and [8], a powerful technique for the evaluation of the Pdimension of a function class satisfying general conditions has appeared and it has been applied with success to cost criteria arising in connection with stabilization problems and H1 control problems in [2] In order to deal with the minimization ....

M. Karpinski and A.J. Macintyre. Polynomial bounds for VC dimension of sigmoidal and general Pfaffian neural networks. J. Comp. Sys. Sci. To appear.

....to the depth of the tree at the input, the length of the computed function in terms of neural network units varies and becomes arbitrary large. Therefore neither the counting argument of [1] nor the method considering the implemented function as a Boolean combination of atomic formulas of [5] can be applied here. In fact the VCdim will be infinite even if only a very small architecture is used. Recall that the VCdim of a class F of functions from a set S to f0; 1g is the largest number k so that there exist values x 1 ; x k 2 S that are shattered by F . A set fx 1 ; x ....

M. Karpinski and A. Macintyre, Polynomial Bounds for VC-dimension of Sigmoidal and General Pfaffian Neural Networks, to appear in Journal of Computer and System Sciences, 1996.

....(for a brief survey see [37] The fact that N computes CD n implies that N 0 shatters S (with regard to different assignments to these s programmable parameters) Thus N 0 has a VC dimension of at least n . On the other hand, the results of Goldberg and Jerrum [20] and Karpinski and Macintyre [29] imply that in this case the number s of programmable parameters in N satisfies n = O(s 2 ) in the case of piecewise polynomial activation functions, respectively n = O(s 4 ) in the case of piecewise exponential activation functions. 2.2 Computation of Functions with Analog Input and Boolean ....

M. Karpinski, and A. Macintyre, "Polynomial bounds for VC-dimension of sigmoidal and general Pfaffian neural networks", to appear in the J. of Comp. and System Sciences.

....(for a brief survey see [34] The fact that N computes CD n implies that N 0 shatters S (with regard to different assignments to these s programmable parameters) Thus N 0 has a VC dimension of at least n . On the other hand the results of Goldberg and Jerrum [18] and Karpinski and Macintyre [26] imply that in this case the number s of programmable parameters in N satisfies n = O(s 2 ) in the case of piecewise polynomial activation functions, respectively n = O(s 4 ) in the case of piecewise exponential activation functions. 2.2 Computation of Functions with Analog Input We have ....

M. Karpinski, and A. Macintyre, "Polynomial bounds for VC-dimension of sigmoidal and general Pfaffian neural networks", to appear in the J. of Comp. and System Sciences.

....network inputs x 2 R n and arbitrary assignments w 2 R m to the weights and biases of these units. We take as the class G of functions considered in the proof of Theorem 5.1 all functions of the form (x; q) z(F (x; w) q) for arbitrary parameters w 2 R m . The results presented in [Karpinski, Macintyre, 1995] imply that the pseudo dimension of this class G of functions is bounded by a polynomial in m, for all common choices of activation functions of the sigmoidal units and all practically relevant density kernels z for the noise process (even involving the exponential function) In the case where ....

M. Karpinski, A. Macintyre, Polynomial bounds for VCdimension of sigmoidal and general Pfaffian neural networks. J. of Computer and System Sciences, to appear.

....infinity results will transfer to this case. For feedforward architectures with w weights the VC dimension is of order O(w log w) if the Heavyside activation function is used [1, 11] If a sigmoidal activation function is used there exist an upper bound of O(w 4 ) and a lower bound of O(w 2 ) [7, 9]. Of course in time series prediction tasks the resulting bounds on the number of training patterns required for valid generalization may be even worse because the training patterns are correlated. 3 Jordan and Elman networks Now we will consider the class of functions computed by a Jordan resp. ....

M. Karpinski and A. Macintyre, Polynomial Bounds for VC-dimension of Sigmoidal and General Pfaffian Neural Networks, to appear in Journal of Computer and System Sciences, 1996.

.... For each A and k, by unfolding the iterations, one may also see the class BA;k as a class of classifiers representable by feedforward neural nets (with k hidden layers ) This trivial fact allows one to easily obtain estimates, based on those bounds which were developed (cf. 3, 1] 6] [7]) for the feedforward case. Theorem 1. For recurrent architectures, with w weights receiving inputs of length k: 1. The VC dimension of threshold recurrent architectures is O(kw log kw) 2. If oe : R R is a fixed piecewise polynomial function, The VC dimension of recurrent architectures with ....

.... 1 for d = 1 and D 2d k 1 for d 2. Proof of Theorem 1 (standard sigmoidal case) By unfolding, the recurrent architecture can be simulated by a feedforward net with kn nodes, where n be the number of nodes in the original architecture, and the same number w of programmable parameters. By [7] there is a O( kn) 2 w 2 ) upper bound on the VC dimension of that architecture. This is O(k 2 w 4 ) as claimed. Note: one can argue that the feedforward net has kw weights, but many of those weights are shared and there are only w n 2w programmable parameters. The result in [7] ....

[Article contains additional citation context not shown here]

M. Karpinski and A. Macintyre, "Polynomial bounds for VC dimension of sigmoidal and general Pfaffian neural networks," J. Computer Sys. Sci., to appear. (Summary in "Polynomial bounds for VC dimension of sigmoidal neural networks," in Proc. 27th ACM Symposium on Theory of Computing, 1995 , pp. 200-208.)

.... For each A and k, by unfolding the iterations, one may also see the class BA;k as a class of classifiers representable by feedforward neural nets (with k hidden layers ) This trivial fact allows one to easily obtain estimates, based on those bounds which were developed (cf. 3, 1] 6] [7]) for the feedforward case. Theorem 1 For recurrent architectures, with w weights receiving inputs of length k: 1. The VC dimension of threshold recurrent architectures is O(kw log kw) 2. If oe : R R is a fixed piecewise polynomial function, The VC dimension of recurrent architectures with ....

.... 1 for d = 1 and D 2d k 1 for d 2. 2 Proof of Theorem 1 (standard sigmoidal case) By unfolding, the recurrent architecture can be simulated by a feedforward net with kn nodes, where n be the number of nodes in the original architecture, and the same number w of programmable parameters. By [7] there is a O( kn) 2 w 2 ) upper bound on the VC dimension of that architecture. This is O(k 2 w 4 ) as claimed. Note: one can argue that the feedforward net has kw weights, but many of those weights are shared and there are only w n 2w programmable parameters. The result in [7] ....

[Article contains additional citation context not shown here]

M. Karpinski and A. Macintyre, "Polynomial bounds for VC dimension of sigmoidal and general Pfaffian neural networks," J. Computer Sys. Sci., to appear. (Summary in "Polynomial bounds for VC dimension of sigmoidal neural networks," in Proc. 27th ACM Symposium on Theory of Computing, 1995 , pp. 200-208.)

....Mathematical Institute, University of Oxford, Oxford OX1 3LB. Research supported in part by a Senior Research Fellowship of the SERC. Email: ajm maths.ox. ac.uk 0 Introduction While working on complexity questions for V C dimension of general neural networks and corresponding semi Pfaffian sets [KM97a], we became convinced that major progress on o minimality should come from a more systematic use of Sard s Theorem and Morse Theory [H76] The importance of these for Khovanski s basic work [K91] is clear. Our result was strongly confirmed by Wilkie s 1996 result [W96] on the o minimality of ....

M. Karpinski and A. Macintyre, Polynomial Bounds for VC Dimension of Sigmoidal and General Pfaffian Neural Networks, J. Comput. System Sci. 54 (1997), pp. 169--176.

....by its smooth functions) then S is closed under complement, is the structure generated by S, and is o minimal. The main action in Wilkie s paper involves his 3.6. This result involves reference to definitions of type 3.5. We observed after reading [W96] our motivation came from our paper [KM97a] in which Sardian arguments abound) that one may modify 3.5 so that the modification of 3.6 remains true under assumptions surely weaker than his DSF. The original 3.5 involved subsets of IR n Theta IR k defined by conditions on (x 1 ; x n ; ffl 1 ; ffl k ) of the form 9x ....

....and the P i are C 1 on each V . Then f V is in Rolle (S) and the graph of f on U is got by closure. In particular, Corollary (Speissegger) Rolle (S) is closed under integration of continuous functions of one variable. 4 Concluding Remarks We came to this topic from our very explicit work [KM97a] on bounds for Vapnik Chervonenkis dimension of semi Pfaffian families. There one made constant appeal to Sardian arguments. The power of the idea there convinced us that a Sardian approach to o minimality would be fruitful. The work of Charbnel and Wilkie certainly confirms this. Our 1996 work ....

M. Karpinski and A. Macintyre, Polynomial Bounds for VC Dimension of Sigmoidal and General Pfaffian Neural Networks, J. Comput. System Sci. 54 (1997), pp. 169--176.

No context found.

Karpinski, M., Macintyre, A. J. 1997. Polynomial bounds for VC dimension of sigmoidal and general Pfa#an neural networks, Journal of Computer and System Sciences, 54:169--176.

No context found.

M. Karpinski and A. Macintyre. Polynomial bounds for VC dimension of sigmoidal and general Pfaffian neural networks. Journal of Computer and System Sciences, 54:169--176, 1997.

No context found.

Marek Karpinski and Angus Macintyre. Polynomial bounds for VC dimension of sigmoidal and general Pfaffian neural networks. Journal of Computer and System Sciences, 54:169--176, 1997.

No context found.

M. Karpinski and A. Macintyre. Polynomial bounds for VC dimension of sigmoidal and general pfaffian neural networks. Journal of Computer and System Sciences, 54:169--176, 1997.

No context found.

M. Karpinski and A. J. Macintyre. Polynomial Bounds for VC Dimension of Sigmoidal and General Pfaffian Neural Networks, Journal of Computer and System Sciences, 54, 1997: 169--176.

No context found.

Marek Karpinski and Angus J. Macintyre, "Polynomial bounds for VC dimension of sigmoidal and general Pfa#an neural networks," J. Comp. Sys. Sci., 54, 169176, 1997. 8

No context found.

Karpinski, M. and A. Macintyre (1997). Polynomial bounds for VC dimension of sigmoidal and general Pfaffian neural networks. Journal of Computer and System Sciences 54, 169--176.

Online articles have much greater impact   More about CiteSeer.IST   Add search form to your site   Submit documents   Feedback  

CiteSeer.IST - Copyright Penn State and NEC