Patrice Simard, Bernard Victorri, Yann LeCun, and John Denker. Tangent prop -- a formalism for specifying selected invariances in an adaptive network. In J. E. Moody, S. J. Hanson, and R. P. Lippmann, editors, Advances in Neural Information Processing Systems 4, pages 895--903, San Mateo, CA, 1992. Morgan Kaufmann.  Home/Search   Document Not in Database   Summary   Related Articles   Check

....Our goal is to make PCA invariant to image plane transformations, while maintaining the clarity and spirit of PCA and without resorting to a complex algorithm. Previous approaches to the problem include EigenTracking [5] that modifies optic flow equations to work with PCA, Tangent Distance [17] and its multi resolution extension [20] where the distance between two images is replaced by the distance between the tangent to the image manifolds, and probabilistic approach [9, 14] that separate images into (a) b) c) d) e) f) Figure 1. Image Coding using PCA depends on correct ....

P. Simard, B. Victorri, Y. Le Cun and J. Denker. Tangent Prop --- a formalism for specifying selected invariances in an adaptive network. In Proceedings of the fourth annual conference NIPS, pages 895-899, Denver, CO., 1991.

....digit recognition, for example, typical invariances include line thickness and image plane translation and rotation. It has been observed that an effective way to make a classifier invariant is to generate synthetic training examples by transforming them according to the desired invariances (cf. [2, 8, 16, 19]) For instance the kernel jittering of [8] performs the synthetic transformations within the matching process between pairs of training examples, thereby effectively matching between two sets of vectors (or between a vector and a set) We could for convenience represent the collection of ....

P. Simard, B. Victorri, T. LeCun, and J. Denker. Tangent prop - a formalism for specifying selected invariances in an adaptive network. In Advances in neural information processing systems 4, 1992.

....digit recognition, for example, typical invariances include line thickness and image plane translation and rotation. It has been observed that an effective way to make a classifier invariant is to generate synthetic training examples by transforming them according to the desired invariances (cf. [2, 8, 16, 19]) For instance the kernel jittering of [8] performs the synthetic transformations within the matching process between pairs of training examples, thereby effectively matching between two sets of vectors (or between a vector and a set) We could for convenience represent the collection of ....

P. Simard, B. Victorri, T. LeCun, and J. Denker. Tangent prop - a formalism for specifying selected invariances in an adaptive network. In Advances in neural information processing systems 4, 1992.

....of using a domain theory for initializing a feedforward neural network have beeen proposed in the literature [16, 46] Prior knowledge can also be used to alter the objective of the hypothesis search space. TangentProp provides explicit knowledge about the derivatives of the function to be learned [35]; it thus overrides backpropagation s bias toward a smooth interpolation between points with explicit training derivatives. Explanation based neural networks use previously trained neural networks as initial domain theories, and compute training derivatives from each observed training sample that ....

Y. L. P.S. Simard, B. Victorri and J. Denker, \Tangentprop - a formalism for specifying selected invariances in an adaptive network," in Advances in Neural Information Processing Systems 4 (J. M. et al., ed.), Morgan Kaufmann, 1992.

....covered by this rule, then iterating this process on the remaining training examples until the rule explains all positive examples. also be used to alter the objective of the hypothesis search space. TangentProp provides explicit knowledge about the derivatives of the function to be learned [10]; it thus overrides backpropagation s bias toward a smooth interpolation between points with explicit training derivatives. Explanation based neural networks (EBNNs) use previously trained neural networks as initial domain theories, and compute training derivatives from each observed training ....

Y. L. P.S. Simard, B. Victorri and J. Denker, \Tangentprop - a formalism for specifying selected invariances in an adaptive network," in Advances in Neural Information Processing Systems 4 (J. M. et al., ed.), Morgan Kaufmann, 1992.

....is used to bias the generalization. Since this bias is knowledgeable, it will partially replace the need for real world experimentation, hence accelerate learning. 8 As Simard and colleagues pointed out, the Back propagation algorithm can be extended to fit target slopes as well as target values [Simard et al. 1992]. Their algorithm tangentprop incrementally updates weights and biases of a neural network such that both the value and the slope error are simultaneously minimized. 3.3 Accommodating Imperfect Action Models Initial experiments with EBNN on a simulated robot navigation task showed a significant ....

Patrice Simard, Bernard Victorri, Yann LeCun, and John Denker. Tangent prop - a formalism for specifying selected invariances in an adaptive network. In J. E. Moody, S. J. Hanson, and R. P. Lippmann, editors, Advances in Neural Information Processing Systems 4, pages 895-903, San Mateo, CA, 1992. Morgan Kaufmann.

....trades off value fit versus slope fit, and that therefore de termines the relative impact of inductive versus analytical components of learning. Gradient descent is employed to iteratively minimize E. Notice that in our implemen tation we used a modified version of the Tangent Prop algorithm [Simard et al. 1992] to refine the weights and biases of the target network [Masuoka, 1993] What is a reasonable method for weighting the contributions of the inductive versus analytical components of learning; that is, for selecting a value for o Because the domain theory might be incorrect, the analytically ....

Patrice Simard, Bernard Victorri, Yann LeCun, and John Denker. Tan- gent prop - a formalism for specifying selected invariances in an adaptive network. In Processing Systems 4, pages 895-903, San Mateo, CA, 1992. Morgan Kaufmann.

....that trades off value fit versus slope fit, and that therefore determines the relative impact of inductive versus analytical components of learning. Gradient descent is employed to iteratively minimize E. Notice that in our implemen tation we used a modified version of the Tangent Prog algorithm [Simard et al. 1992] to refine the weights and biases of the target network [Masuoka, 1993] What is a reasonable method for weighting the contributions of the inductive versus analytical components of learning; that is, for selecting a value for o Because the domain theory might be incorrect, the analytically ....

Patrice Simard, Bernard Victorri, Yann LeCun, and John Denker. Tangent prop - a formalism for specifying selected invariances in an adaptive network. In J. E. Moody, S. J. Hanson, and R. P. Lippmann, editors, Advances in Neural Information Processing Systems 4, pages 895-903, San Mateo, CA, 1992. Morgan Kaufmann.

....which takes both value error, Ewe, and slope error, slope, weighted by a gain parameter c into account. Target values are fitted using the Backpropagation algorithm [17] Target slopes are fitted using the Tangent Prop algorithm, which is an analogue of Backpropagation for fitting slopes [20]. Clearly, slopes extracted by EBNN can be wrong. This is because they are computed using artificial neural networks, which themselves are constructed from training examples. Consequently, target slopes can mislead the generalization. EBNN provides a simple but effective mechanism to recover from ....

Patrice Simard, Bernard Victorri, Yann LeCun, and John Denker. Tangent prop a formalism for specifying selected invariances in an adaptive network. In J. E. Moody, S. J. Hanson, and R. P. Lippmann, editors, Advances in Neural Information Processing Systems 4, pages 895 903, San Mateo, CA, 1992. Morgan Kauf- mann.

....approach is to constrain training by forcing the network to fit target slopes, or derivatives, of the learned function, in addition to the usual constraint that it must fit target values of the function. This approach of constraining networks to fit target slopes has been used, for example, in [ Simard et al. 1992 ] to constrain a character recognition network to produce outputs that are invariant to certain transformations of the inputs (e.g. rotation of the input character) It has also been used in the Explanation Based Neural Network (EBNN) alogorithm ( Mitchell and Thrun, 1993 ] which uses prior ....

....encoded in previously learned networks to derive slope constraints that must be satisfied by the function to be learned. The goal of this paper is to explore the use of slope information for constraining neural network backpropagation learning. We describe tangent prop algorithm presented in [ Simard et al. 1992 ] somewhat in a more general and detailed way for the later analysis. We examine the role of slope constraints in reducing the generalization error in learning, as well as the complexity of these algorithms and their robustness to errors in the training data. A combination of analytical and ....

[Article contains additional citation context not shown here]

Patrice Simard, Bernard Victorri, Yann LeCun, and John Denker. Tangent prop -- a formalism for specifying selected invariances in an adaptive network. In J. E. Moody, S. J. Hanson, and R. P. Lippmann, editors, Advances in Neural Information Processing Systems 4, pages 895--903. Morgan Kaufmann, San Mateo, CA, 1992. 41

....tangent distance of [56] or a deformable matching technique as in [25] These methods carry the disadvantages of being invariant only to small deformations of the sample data and of being computationally very expensive. The computational cost might be overcome by using the tangent prop algorithm [57]. The tangent prop algorithm incorporates the tangent direction of each training sample within the backpropagation algorithm to learn the function that describes the object (e.g. face) to be learned. This allows better approximations of the function to be learned than when one only uses the ....

....as precisely as the algorithm described above, but would be computationally less demanding. It has been shown that, as the number of samples increase, the tangent prop algorithm reduces its error on test data quicker than the backpropagation algorithm for the recognition of hand written digits [57]. 3.2 The eigenspace representation Since the feature space corresponds to a dense (pixel) feature representation, its dimensionality is expected to be too large to allow the computation of the subspaces described above (Gaussians or mixture of Gaussians) For example, if face images are 100 by ....

[Article contains additional citation context not shown here]

P. Simard, B. Victorri, Y. LeCun and J. Denker, "Tangent Prop -- A formalism for specifying selected invariances in an adaptative network," In Advances in Neural Information Processing Systems, 4, J.E. Moody, S.J. Hanson and R.P. Lippmann, eds., pp. 651-655, Morgan Kaufmann, 1992.

....constraints. A second group of constraint learning approaches learns slope constraints. For each data point, previously acquired knowledge is used to constrain the slope of the function to be learned. The idea of incorporating slopes into neural network learning is due to Simard and colleagues [57], who proposed to encode certain (known) invariances in character recognition by directional slope information. The EBNN [66, 67] algorithm extends their approach in that it learns the slope constraints. EBNN rests on the same assumption as the recursive functional decomposition approaches listed ....

P. Simard, B. Victorri, Y. LeCun, and J. Denker. Tangent prop -- a formalism for specifying selected invariances in an adaptive network. In J. E. Moody, S. J. Hanson, and R. P. Lippmann, editors, Advances in Neural Information Processing Systems 4, pages 895--903, San Mateo, CA, 1992. Morgan Kaufmann.

....1. Introduction The design of a classifier that is invariant w.r.t. certain transformations is an important aspect in pattern recognition. Many approaches to invariant pattern recognition are known [1] among them an invariant distance measure called tangent distance. TD was proposed in [2, 3] and proved to be very effective in the domain of optical character recognition. Distance measures like the Euclidean distance and related ones are very sensitive to small transformations, even though these transformations do not affect class membership. In contrast to that, TD is able to ....

P. Simard, B. Victorri, Y. Le Cun, and J. Denker, "Tangent prop -- A formalism for specifying selected invariances in an adaptive network," in Advances in Neural Information Proc. Systems, vol. 4, pp. 895--903, Morgan Kaufmann Publishers, Inc., 1992.

....line locally. If K types of continuous transformation are applied, the manifold will be K dimensional. Linear approximations of the transformation manifold have been used to significantly improve the performance of supervised classifiers such as nearest neighbors [21] and multilayer perceptrons [22]. Linear generative models (factor analysis, mixtures of factor analysis) have also been modified using linear approximations of the transformation manifold to build in some degree of transformation invariance [23] In general, the linear approximation is accurate for transformations that couple ....

P. Y. Simard, B. Victorri, Y. Le Cun, and J. Denker, "Tangent prop -- a formalism for specifying selected invariances in an adaptive network," in Advances in Neural Information Processing Systems 4. Morgan Kaufmann, San Mateo CA., 1992.

....although the curve can be approximated by a straight line locally. Linear approximations of the transformation manifold have been used to significantly improve the performance of feedforward discriminative classi ers such as nearest neighbors (Simard et al. 1993) and multilayer perceptrons (Simard et al. 1992). Linear generative models (factor analysis, mixtures of factor analysis) have also been modi ed using linear approximations of the transformation manifold to build in some degree of transformation invariance (Hinton et al. 1997) In general, the linear approximation is accurate for ....

P. Y. Simard, B. Victorri, Y. Le Cun and J. Denker 1992. Tangent Prop { A formalism for specifying selected invariances in an adaptive network. In Advances in Neural Information Processing Systems 4, Morgan Kaufmann, San Mateo, CA.

....In particular, they proved that SVM are a limiting case of their Restricted Bayesian Classifiers. We extend Tong s and Koller s result by showing that VRM subsumes several well known techniques such as Ridge Regression (Hoerl and Kennard, 1970) Constrained Logistic Classifier, or Tangent Prop (Simard et al. 1992). We then go on to show how VRM naturally leads to simple algo rithms that can deal with problems for which one would have formally considered purely generative models. We provide algorithms and preliminary empirical results for dealing with unlabeled data or recognizing classes with very ....

....In the limit, this is equivalent to replacing each initial example by a distribution whose shape represents the desired invariances. This formulation naturally leads to a special case of VRM in which the local density estimates P x i (x) are elongated in the direction of invariance. Tangent Prop (Simard et al. 1992) is a more sophisticated way to incorporate invariances by adding an adequate regularization term to the cost function. Tangent Prop has been formally proved to be equivalent to generating synthetic examples with infinitesimal deformations (Leen, 1995) This analysis makes Tangent Prop a special ....

Simard, P., Victorri, B., Le Cun, Y., and Denker, J. (1992). Tangent prop: a formalism for specifying selected invariances in adaptive networks. In Advances in Neural Information Processing Systems 4, Denver, CO. Morgan Kaufman.

....features. Examples are the nearest neighbor classi er using tangent distance [152] and deformable template matching [84] These approaches only achieve invariance to small amounts of linear transformations and nonlinear deformations. Besides, they are computationally very intensive. Simard et al. [153] proposed an algorithm named Tangent Prop to minimize the derivative of the classi er outputs with respect to distortion parameters, i.e. to improve the invariance property of the classi er to the selected distortion. This 83 makes the trained classi er computationally very ecient. 9.2 ....

P. Simard, B. Victorri, Y. LeCun, and J. Denker, \Tangent prop - a formalism for specifying selected invariances in an adaptive network," in Advances in Neural Information Processing Systems 4, J. E. Moody, S. J. Hanson, and R. P. Lippmann, eds., pp. 651-655, Morgan Kaufmann, CA, 1992. 100

.... E u = w or E v = w in O(1) times the time required to calculate the normal gradient. Thus, J prop is suitable for training models to have specific first derivatives, or for implementing several other well known algorithms such as Double Backpropagation (Drucker Le Cun, 1992) and Tangent Prop (Simard, Victorri, Le Cun Denker, 1992). Suitable choice of a, b, u and v make the functions E u and E v surprisingly expressive. For example, with u set to 1 The Calculus of Jacobian Adaptation Flake a unit vector a can be interpreted as a target value for a particular row of the Jacobian matrix. As will be shown, other ....

....with noise) but it explicitly penalizes large first derivatives of the model (unlike weight decay) Double Backpropagation can be seen as a special case of J prop, the algorithm derived in the next section. As to the general problem of coercing the first derivatives of a model to specific values, (Simard, Victorri, Le Cun Denker, 1992) introduced the Tangent Prop algorithm, which was used to train MLPs for optical character recognition to be insensitive to small affine transformations in the character space. Tangent Prop is very similar to the algorithm proposed 2 The Calculus of Jacobian Adaptation Flake in this paper but ....

[Article contains additional citation context not shown here]

Simard, P., Victorri, B., Le Cun, Y., & Denker, J. (1992). Tangent prop---A formalism for specifying selected invariances in an adaptive network. In Moody, J. E., Hanson, S. J., & Lippmann, R. P. (Eds.), Advances in Neural Information Processing Systems, volume 4, (pp. 895--903).

No context found.

Patrice Simard, Bernard Victorri, Yann LeCun, and John Denker. Tangent prop -- a formalism for specifying selected invariances in an adaptive network. In J. E. Moody, S. J. Hanson, and R. P. Lippmann, editors, Advances in Neural Information Processing Systems 4, pages 895--903, San Mateo, CA, 1992. Morgan Kaufmann.

No context found.

Patrice Simard, Bernard Victorri, Yann LeCun, and John Denker. Tangent prop -- a formalism for specifying selected invariances in an adaptive network. In Advances in Neural Information Processing Systems 4, J. E. Moody and S. J. Hanson and R. P. Lipmann, Ed., Morgan Kaufmann, 1992, pp. 895--903.

No context found.

P. Simard, B. Victorri, Y. Le Cun and J. Denker. Tangent Prop --- a formalism for specifying selected invariances in an adaptive network. In Proceedings of the fourth annual conference NIPS, pages 895-899, Denver, CO., 1991.

No context found.

P. Simard, B. Victorri, Y. Le Cun, and J. Denker. Tangent prop---a formalism for specifying selected invariances in an adaptive network. In J. E. Moody, S. J. Hanson, and R. P. Lippmann, editors, Advances in Neural Information Processing Systems 4, pages 895--903. Morgan Kaufmann, San Mateo, CA, 1992.

No context found.

Simard, P, Victorri, B, LeCun, Y & Denker, J (1992). Tangent Prop - A formalism for specifying selected invariances in an adaptive network. In JE Moody, SJ Hanson & RP Lippmann, editors, Advances in Neural Information Processing Systems 4. San Mateo, CA: Morgan Kaufmann.

No context found.

P. Simard, B. Victorri, T. LeCun, and J. Denker. Tangent prop - a formalism for specifying selected invariances in an adaptive network. In Advances in neural information processing systems 4, 1992.

No context found.

Patrice Simard, Bernard Victorri, Yann Le Cun, and John Denker. Tangent Prop - a formalism for specifying selected invariances in an adaptive network. Advances in Neural Information Processing

First 50 documents  Next 50

Online articles have much greater impact   More about CiteSeer.IST   Add search form to your site   Submit documents   Feedback  

CiteSeer.IST - Copyright Penn State and NEC