Abstract

Covariance matrices are important in many areas of neural modelling. In Hopfield networks they are used to form the weight matrix which controls the autoassociative properties of the network. In Gaussian processes, which have been shown to be the infinite neuron limit of many regularised feedforward neural networks, covariance matrices control the form of Bayesian prior distribution over function space. This thesis examines interesting modifications to the standard covariance matrix methods to increase functionality or efficiency of these neural techniques. Firstly the problem of adapting Gaussian process priors to perform regression on switching regimes is tackled. This involves the use of block covariance matrices and Gibbs sampling methods. Then the use of Toeplitz methods is proposed for Gaussian process regression where sampling positions can be chosen. A comparison is made between Hopfield weight matrices, and sample covariances. This allows work on sample covariances to be used ...

Citations

1529	Introduction to the Theory of Neural Computation - Hertz, Krogh, et al. - 1991
1523	Matrix Computations - Golub, Loan - 1983
1298	Neural networks and physical systems with emergent collective computational abilities - Hopfield - 1982
989	Spline models for observational data - Wahba - 1990
976	Multilayer feedforward networks are universal approximators - Hornik, Stinchcombe, et al. - 1989
612	A view of the EM algorithm that justifies incremental, sparse, and other variants - Neal, Hinton - 1998
584	Learning representations by back-propagation errors - Rumelhart, Hinton, et al. - 1986
530	Analog VLSI and Neural Systems - Mead - 1989
436	Neural” computation of decision optimization problems - Hopfield, Tank - 1985
346	A practical Bayesian framework for backpropagation networks.Neural computation - MacKay
236	Probability Theory - Loéve - 1955
223	Probability and random processes - Grimmett, Stirzaker - 1982
158	Modeling Brain Function - Amit - 1989
119	Evaluation of Gaussian Processes and other Methods for Non-Linear Regression - Rasmussen - 1996
106	Monte Carlo Implementation of Gaussian Process Models for Bayesian Regression and Classification - Neal - 1997
104	Pastur, “Distributions of eigenvalues of some sets of random matrices - Mar�enko, A - 2012
103	Bayesian classification with Gaussian processes - Williams, Barber - 1998
88	Improper Priors, Spline Smoothing and the Problem of Guarding Against Model Errors in Regression - Wahba - 1978
78	The capacity of the Hopfield associative memory - McEliece, Posner, et al. - 1987
77	Learning algorithms with optimal stability in neural networks - Krauth, Mezard - 1987
61	Efficient implementation of Gaussian processes”, draft manuscript - Gibbs, MacKay - 1997
60	Statistical mechanics of neural networks near saturation - Amit, Gutfeund, et al. - 1987
58	Bayesian regularization and pruning using a Laplace prior - Williams - 1995
54	Learning in neural networks with material synapses - Amit, Fusi - 1994
48	An algorithm for the inversion of finite Toeplitz matrices - Trench - 1964
45	The strong limits of random matrix spectra for sample matrices of independent elements - Wachter - 1978
43	Gaussian Processes: A Replacement for Supervised Neural Networks? Lecture notes for a tutorial at NIPS - MacKay - 1997
31	Statistical neurodynamics of associative memory - Amari, Maginu - 1988
30	Collective computational properties of neural networks: New learning mechanisms, Phys. Rev. A vol 34 - Personnaz, Guyon, et al. - 1986
28	Gaussian processes for Bayesian classification via hybrid Monte Carlo - Barber, Williams - 1997
28	Associative recall of memory without errors - Kanter - 1987
26	Regression with input-dependent noise: A Gaussian process treatment - Goldberg, Williams, et al. - 1998
24	Variational Gaussian process classifiers - Gibbs, MacKay
23	A memory which forgets - Parisi - 1986
21	Priors for infinite networks - Neal - 1994
20	Bayesian numerical analysis - Skilling - 1993
20	1974], The Existence of Persistent States - Little
20	Network of formal neurons and memory palimpsests, Europhys - Nadal, Toulouse, et al. - 1986
19	On the use of evidence in neural networks - Wolpert - 1993
19	A logical calculus of the ideas immanent in neural nets - McCulloch, Pitts - 1943
18	Spectral analysis of networks with random topologies - Grenander, Silverstein - 1977
18	Iterative retrieval of sparsely coded associative memory patterns“, Neural Networks 9 - Schwenker, Sommer, et al. - 1996
17	Circulant preconditioned Toeplitz Least Squares Iterations - Chan, Nagy, et al. - 1989
15	Prediction with Gaussian processes - Williams - 1999
15	On curve fitting and optimal design for regression - O’Hagan - 1978
13	Hyperparameters: Optimise, or integrate out - MacKay - 1993
13	Solvable models of working memories - M'ezard, Nadal, et al. - 1986
12	Matrix methods for engineers and scientists - Barnett - 1979
12	Information storage and retrieval in spin-glass like neural networks - Personnaz, Guyen, et al. - 1985
10	Nonuniversal critical dynamics in Monte-Carlo simulations,” Phys - Swendsen, Wang - 1987