Results 1  10
of
19
On the capabilities of neural networks using limited precision weights
 Neural Netw
, 2002
"... This paper analyzes some aspects of the computational power of neural networks using integer weights in a very restricted range. Using limited range integer values opens the road for efficient VLSI implementations because i) a limited range for the weights can be translated into reduced storage req ..."
Abstract

Cited by 14 (0 self)
 Add to MetaCart
(Show Context)
This paper analyzes some aspects of the computational power of neural networks using integer weights in a very restricted range. Using limited range integer values opens the road for efficient VLSI implementations because i) a limited range for the weights can be translated into reduced storage requirements and ii) integer computation can be implemented in a more efficient way than the floating point one. The paper concentrates on classification problems and shows that, if the weights are restricted in a drastic way (both range and precision), the existence of a solution is not to be taken for granted anymore. The paper presents an existence result which relates the difficulty of the problem as characterized by the minimum distance between patterns of different classes to the weight range necessary to ensure that a solution exists. This result allows us to calculate a weight range for a given category of problems and be confident that the network has the capability to solve the given problems with integer weights in that range. Worstcase lower bounds are given for the number of entropy bits and weights necessary to solve a given problem. Various practical issues such as the relationship between the information entropy bits and storage bits are also discussed. The approach presented here uses a worstcase analysis. Therefore, the approach tends to overestimate the values obtained for the weight range, the number of bits and the number of weights. The paper also presents some statistical considerations that can be used to give up the absolute confidence of a successful training in exchange for values more appropriate for practical use. The approach presented is also discussed in the context of the VCcomplexity.
Toward a Miniature Wireless Integrated Multisensor Microsystem for Industrial and Biomedical Applications
 Industrial and Biomedical Applications, IEEE Sensors Journal: Special Issue on Integrated Multisensor Systems and Signal Processing
, 2002
"... This paper presents our work toward the integration of a multisensor microsystem with wireless communication, using systemonchip (SoC) methodology. Four different forms of microelectronic sensors have been fabricated on two separate 5 5mm 2 silicon chips measuring pH, conductivity, dissolved oxy ..."
Abstract

Cited by 6 (3 self)
 Add to MetaCart
This paper presents our work toward the integration of a multisensor microsystem with wireless communication, using systemonchip (SoC) methodology. Four different forms of microelectronic sensors have been fabricated on two separate 5 5mm 2 silicon chips measuring pH, conductivity, dissolved oxygen concentration, and temperature. The sensors are integrated with a sensor fusion chip comprising analog circuitry for sensor operation and signal amplification prior to digital decoding and transmission. The microsystem prototype will be packaged in a miniature capsule, which measures 16 mm 55 mm including batteries and dissipates 6.3 mW for a minimal life cycle of 12 h. Index TermsLaboratoryonachip (LoC), microsystem, multisensor array, systemonchip (SoC), wireless communication.
An integrated mixedmode neural network architecture for megasynapse ANNs
 In Proceedings of the 2002 International Joint Conference on Neural Networks IJCNN’02
, 2002
"... Abstract This paper presents a new VLSI architecture for ANNs based on the combination of digital signalling and analog computing. It achieves a high level of parallelism as well as ecient area and power usage making very large networks possible. An implementation is presented combining 33k synap ..."
Abstract

Cited by 4 (2 self)
 Add to MetaCart
(Show Context)
Abstract This paper presents a new VLSI architecture for ANNs based on the combination of digital signalling and analog computing. It achieves a high level of parallelism as well as ecient area and power usage making very large networks possible. An implementation is presented combining 33k synapses and 256 neurons on 9 mm2 of silicon area. I.
Quantization and Pruning of Multilayer Perceptrons: Towards Compact Neural Networks
, 1997
"... A connectionist system or neural network is a massively parallel network of weighted interconnections, which connect one or more layers of nonlinear processing elements (neurons). To fully profit from the inherent parallel processing of these networks, development of parallel hardware implementatio ..."
Abstract

Cited by 2 (1 self)
 Add to MetaCart
A connectionist system or neural network is a massively parallel network of weighted interconnections, which connect one or more layers of nonlinear processing elements (neurons). To fully profit from the inherent parallel processing of these networks, development of parallel hardware implementations is essential. However, these hardware implementations often differ in various ways from the ideal mathematical description of a neural network model. It is, for example, required to have quantized network parameters, in both electronic and optical implementations of neural networks. This can be because device operation is quantized or a coarse quantization of network parameters is beneficial for designing compact networks. Most of the standard algorithms for training neural networks are not suitable for quantized networks because they are based on gradient descent and require a high accuracy of the network parameters Several weight discretization techniques have been developed to reduce the required accuracy further without deterioration of network performance. One of the earliest of these techniques [Fiesler88] is further investigated and improved in this report. Another way to obtain compact networks is by minimizing their topology for the problem at hand. However, it is impossible to know a priori the size of such minimal network topology. An unsuitable
Expectation backpropagation: Parameterfree training of multilayer neural networks with continuous or discrete weights
 In Advances in Neural Information Processing Systems (NIPS
, 2014
"... Multilayer Neural Networks (MNNs) are commonly trained using gradient descentbased methods, such as BackPropagation (BP). Inference in probabilistic graphical models is often done using variational Bayes methods, such as Expectation Propagation (EP). We show how an EP based approach can also be us ..."
Abstract

Cited by 2 (1 self)
 Add to MetaCart
(Show Context)
Multilayer Neural Networks (MNNs) are commonly trained using gradient descentbased methods, such as BackPropagation (BP). Inference in probabilistic graphical models is often done using variational Bayes methods, such as Expectation Propagation (EP). We show how an EP based approach can also be used to train deterministic MNNs. Specifically, we approximate the posterior of the weights given the data using a “meanfield ” factorized distribution, in an online setting. Using online EP and the central limit theorem we find an analytical approximation to the Bayes update of this posterior, as well as the resulting Bayes estimates of the weights and outputs. Despite a different origin, the resulting algorithm, Expectation BackPropagation (EBP), is very similar to BP in form and efficiency. However, it has several additional advantages: (1) Training is parameterfree, given initial conditions (prior) and the MNN architecture. This is useful for largescale problems, where parameter tuning is a major challenge. (2) The weights can be restricted to have discrete values. This is especially useful for implementing trained MNNs in precision limited hardware chips, thus improving their speed and energy efficiency by several orders of magnitude. We test the EBP algorithm numerically in eight binary text classification tasks. In all tasks, EBP outperforms: (1) standard BP with the optimal constant learning rate (2) previously reported state of the art. Interestingly, EBPtrained MNNs with binary weights usually perform better than MNNs with continuous (real) weights if we average the MNN output using the inferred posterior. 1
Backpropagation for energyefficient neuromorphic computing.
 In Advances in Neural Information Processing Systems,
, 2015
"... Abstract Solving real world problems with embedded neural networks requires both training algorithms that achieve high performance and compatible hardware that runs in real time while remaining energy efficient. For the former, deep learning using backpropagation has recently achieved a string of s ..."
Abstract

Cited by 1 (0 self)
 Add to MetaCart
(Show Context)
Abstract Solving real world problems with embedded neural networks requires both training algorithms that achieve high performance and compatible hardware that runs in real time while remaining energy efficient. For the former, deep learning using backpropagation has recently achieved a string of successes across many domains and datasets. For the latter, neuromorphic chips that run spiking neural networks have recently achieved unprecedented energy efficiency. To bring these two advances together, we must first resolve the incompatibility between backpropagation, which uses continuousoutput neurons and synaptic weights, and neuromorphic designs, which employ spiking neurons and discrete synapses. Our approach is to treat spikes and discrete synapses as continuous probabilities, which allows training the network using standard backpropagation. The trained network naturally maps to neuromorphic hardware by sampling the probabilities to create one or more networks, which are merged using ensemble averaging. To demonstrate, we trained a sparsely connected network that runs on the TrueNorth chip using the MNIST dataset. With a high performance network (ensemble of 64), we achieve 99.42% accuracy at 108 µJ per image, and with a high efficiency network (ensemble of 1) we achieve 92.7% accuracy at 0.268 µJ per image.
Implementing neural models in silicon
, 2004
"... Neural models are used in both computational neuroscience and in pattern recognition. The aim of the first is understanding of real neural systems, and of the second is gaining better, possibly brainlike performance for systems being built. In both cases, the highly parallel nature of the neural sy ..."
Abstract

Cited by 1 (0 self)
 Add to MetaCart
(Show Context)
Neural models are used in both computational neuroscience and in pattern recognition. The aim of the first is understanding of real neural systems, and of the second is gaining better, possibly brainlike performance for systems being built. In both cases, the highly parallel nature of the neural system contrasts with the sequential nature of computer systems, resulting in slow and complex simulation software. More direct implementation in hardware (whether digital or analogue) holds out the promise of faster emulation both because hardware implementation is inherently faster than software, and because the operation is much more parallel. There are costs to this: modifying the system (for example to test out variants of the system) is much harder when a full application specific integrated circuit has been built. Fast emulation can permit direct incorporation of a neural model into a system, permitting realtime input and output. Appropriate selection of implementation technology can help to make interfacing the system to external devices simpler. We review the technologies involved, and discuss some example systems. 1 Why implement neural models in silicon? There are two primary reasons for implementing neural models: one is to attempt to gain better, and possibly
A Mixed AnalogDigital Artificial Neural Network Architecture with OnChip Learning
, 1998
"... This paper presents a novel artificial neural network architecture with onchip learning capability. The issue of straightforward designflow integration of an autonomous unit is addressed with a mixed analogdigital approach, by implementing a chargebased artificial neural network which interacts ..."
Abstract

Cited by 1 (1 self)
 Add to MetaCart
This paper presents a novel artificial neural network architecture with onchip learning capability. The issue of straightforward designflow integration of an autonomous unit is addressed with a mixed analogdigital approach, by implementing a chargebased artificial neural network which interacts with digital control and processing units. We demonstrate the circuit architecture and designflow approach for the case of a Hamming network performing pixelpattern recognition. Keywords Chargebased ANN, mixedmode ANN hardware architecture, ANN integration designflow. I. Introduction T HE ABILITY of artificial neural networks (ANN) to acquire knowledge of their surrounding environment and adapt to it, as well as their use of a high degree of computing parallelism makes them very efficient in many application fields including process and quality control, consumer products, optical character and speech recognition, and complex forecasting tasks among many others [1]. Silicon implemen...
Mean Field Bayes Backpropagation: scalable training of multilayer neural networks with binary weights
, 2013
"... Significant success has been reported recently ucsing deep neural networks for classification. Such large networks can be computationally intensive, even after training is over. Implementing these trained networks in hardware chips with a limited precision of synaptic weights may improve their speed ..."
Abstract

Cited by 1 (0 self)
 Add to MetaCart
(Show Context)
Significant success has been reported recently ucsing deep neural networks for classification. Such large networks can be computationally intensive, even after training is over. Implementing these trained networks in hardware chips with a limited precision of synaptic weights may improve their speed and energy efficiency by several orders of magnitude, thus enabling their integration into small and lowpower electronic devices. With this motivation, we develop a computationally efficient learning algorithm for multilayer neural networks with binary weights, assuming all the hidden neurons have a fanout of one. This algorithm, derived within a Bayesian probabilistic online setting, is shown to work well for both synthetic and realworld problems, performing comparably to algorithms with realvalued weights, while retaining computational tractability. 1
EURASIP Journal on Applied Signal Processing 2005:7, 993–1004 c ○ 2005 Hindawi Publishing Corporation Object Recognition SystemonChip Using the Support Vector Machines
, 2004
"... The first aim of this work is to propose the design of a systemonchip (SoC) platform dedicated to digital image and signal processing, which is tuned to implement efficiently multiplyandaccumulate (MAC) vector/matrix operations. The second aim of this work is to implement a recent promising neur ..."
Abstract
 Add to MetaCart
The first aim of this work is to propose the design of a systemonchip (SoC) platform dedicated to digital image and signal processing, which is tuned to implement efficiently multiplyandaccumulate (MAC) vector/matrix operations. The second aim of this work is to implement a recent promising neural network method, namely, the support vector machine (SVM) used for realtime object recognition, in order to build a vision machine. With such a reconfigurable and programmable SoC platform, it is possible to implement any SVM function dedicated to any object recognition problem. The final aim is to obtain an automatic reconfiguration of the SoC platform, based on the results of the learning phase on an objects ’ database, which makes it possible to recognize practically any object without manual programming. Recognition can be of any kind that is from image to signal data. Such a system is a generalpurpose automatic classifier. Many applications can be considered as a classification problem, but are usually treated specifically in order to optimize the cost of the implemented solution. The cost of our approach is more important than a dedicated one, but in a near future, hundreds of millions of gates will be common and affordable compared to