Results 1 - 10
of
19
On the capabilities of neural networks using limited precision weights
- Neural Netw
, 2002
"... This paper analyzes some aspects of the computational power of neural networks using integer weights in a very restricted range. Using limited range integer values opens the road for efficient VLSI implementations because i) a limited range for the weights can be translated into reduced storage req ..."
Abstract
-
Cited by 14 (0 self)
- Add to MetaCart
(Show Context)
This paper analyzes some aspects of the computational power of neural networks using integer weights in a very restricted range. Using limited range integer values opens the road for efficient VLSI implementations because i) a limited range for the weights can be translated into reduced storage requirements and ii) integer computation can be implemented in a more efficient way than the floating point one. The paper concentrates on classification problems and shows that, if the weights are restricted in a drastic way (both range and precision), the existence of a solution is not to be taken for granted anymore. The paper presents an existence result which relates the difficulty of the problem as characterized by the minimum distance between patterns of different classes to the weight range necessary to ensure that a solution exists. This result allows us to calculate a weight range for a given category of problems and be confident that the network has the capability to solve the given problems with integer weights in that range. Worst-case lower bounds are given for the number of entropy bits and weights necessary to solve a given problem. Various practical issues such as the relationship between the information entropy bits and storage bits are also discussed. The approach presented here uses a worst-case analysis. Therefore, the approach tends to overestimate the values obtained for the weight range, the number of bits and the number of weights. The paper also presents some statistical considerations that can be used to give up the absolute confidence of a successful training in exchange for values more appropriate for practical use. The approach presented is also discussed in the context of the VC-complexity.
Toward a Miniature Wireless Integrated Multisensor Microsystem for Industrial and Biomedical Applications
- Industrial and Biomedical Applications, IEEE Sensors Journal: Special Issue on Integrated Multisensor Systems and Signal Processing
, 2002
"... This paper presents our work toward the integration of a multisensor microsystem with wireless communication, using system-on-chip (SoC) methodology. Four different forms of microelectronic sensors have been fabricated on two separate 5 5mm 2 silicon chips measuring pH, conductivity, dissolved oxy ..."
Abstract
-
Cited by 6 (3 self)
- Add to MetaCart
This paper presents our work toward the integration of a multisensor microsystem with wireless communication, using system-on-chip (SoC) methodology. Four different forms of microelectronic sensors have been fabricated on two separate 5 5mm 2 silicon chips measuring pH, conductivity, dissolved oxygen concentration, and temperature. The sensors are integrated with a sensor fusion chip comprising analog circuitry for sensor operation and signal amplification prior to digital decoding and transmission. The microsystem prototype will be packaged in a miniature capsule, which measures 16 mm 55 mm including batteries and dissipates 6.3 mW for a minimal life cycle of 12 h. Index Terms---Laboratory-on-a-chip (LoC), microsystem, multisensor array, system-on-chip (SoC), wireless communication.
An integrated mixed-mode neural network architecture for megasynapse ANNs
- In Proceedings of the 2002 International Joint Conference on Neural Networks IJCNN’02
, 2002
"... Abstract- This paper presents a new VLSI archi-tecture for ANNs based on the combination of digital signalling and analog computing. It achieves a high level of parallelism as well as ecient area and power usage making very large networks possible. An imple-mentation is presented combining 33k synap ..."
Abstract
-
Cited by 4 (2 self)
- Add to MetaCart
(Show Context)
Abstract- This paper presents a new VLSI archi-tecture for ANNs based on the combination of digital signalling and analog computing. It achieves a high level of parallelism as well as ecient area and power usage making very large networks possible. An imple-mentation is presented combining 33k synapses and 256 neurons on 9 mm2 of silicon area. I.
Quantization and Pruning of Multilayer Perceptrons: Towards Compact Neural Networks
, 1997
"... A connectionist system or neural network is a massively parallel network of weighted interconnections, which connect one or more layers of non-linear processing elements (neurons). To fully profit from the inherent parallel processing of these networks, development of parallel hardware implementatio ..."
Abstract
-
Cited by 2 (1 self)
- Add to MetaCart
A connectionist system or neural network is a massively parallel network of weighted interconnections, which connect one or more layers of non-linear processing elements (neurons). To fully profit from the inherent parallel processing of these networks, development of parallel hardware implementations is essential. However, these hardware implementations often differ in various ways from the ideal mathematical description of a neural network model. It is, for example, required to have quantized network parameters, in both electronic and optical implementations of neural networks. This can be because device operation is quantized or a coarse quantization of network parameters is beneficial for designing compact networks. Most of the standard algorithms for training neural networks are not suitable for quantized networks because they are based on gradient descent and require a high accuracy of the network parameters Several weight discretization techniques have been developed to reduce the required accuracy further without deterioration of network performance. One of the earliest of these techniques [Fiesler-88] is further investigated and improved in this report. Another way to obtain compact networks is by minimizing their topology for the problem at hand. However, it is impossible to know a priori the size of such minimal network topology. An unsuitable
Expectation backpropagation: Parameter-free training of multilayer neural networks with continuous or discrete weights
- In Advances in Neural Information Processing Systems (NIPS
, 2014
"... Multilayer Neural Networks (MNNs) are commonly trained using gradient descent-based methods, such as BackPropagation (BP). Inference in probabilistic graphical models is often done using variational Bayes methods, such as Expec-tation Propagation (EP). We show how an EP based approach can also be us ..."
Abstract
-
Cited by 2 (1 self)
- Add to MetaCart
(Show Context)
Multilayer Neural Networks (MNNs) are commonly trained using gradient descent-based methods, such as BackPropagation (BP). Inference in probabilistic graphical models is often done using variational Bayes methods, such as Expec-tation Propagation (EP). We show how an EP based approach can also be used to train deterministic MNNs. Specifically, we approximate the posterior of the weights given the data using a “mean-field ” factorized distribution, in an online setting. Using online EP and the central limit theorem we find an analytical ap-proximation to the Bayes update of this posterior, as well as the resulting Bayes estimates of the weights and outputs. Despite a different origin, the resulting algorithm, Expectation BackPropagation (EBP), is very similar to BP in form and efficiency. However, it has several addi-tional advantages: (1) Training is parameter-free, given initial conditions (prior) and the MNN architecture. This is useful for large-scale problems, where param-eter tuning is a major challenge. (2) The weights can be restricted to have discrete values. This is especially useful for implementing trained MNNs in precision lim-ited hardware chips, thus improving their speed and energy efficiency by several orders of magnitude. We test the EBP algorithm numerically in eight binary text classification tasks. In all tasks, EBP outperforms: (1) standard BP with the optimal constant learning rate (2) previously reported state of the art. Interestingly, EBP-trained MNNs with binary weights usually perform better than MNNs with continuous (real) weights- if we average the MNN output using the inferred posterior. 1
Backpropagation for energy-efficient neuromorphic computing.
- In Advances in Neural Information Processing Systems,
, 2015
"... Abstract Solving real world problems with embedded neural networks requires both training algorithms that achieve high performance and compatible hardware that runs in real time while remaining energy efficient. For the former, deep learning using backpropagation has recently achieved a string of s ..."
Abstract
-
Cited by 1 (0 self)
- Add to MetaCart
(Show Context)
Abstract Solving real world problems with embedded neural networks requires both training algorithms that achieve high performance and compatible hardware that runs in real time while remaining energy efficient. For the former, deep learning using backpropagation has recently achieved a string of successes across many domains and datasets. For the latter, neuromorphic chips that run spiking neural networks have recently achieved unprecedented energy efficiency. To bring these two advances together, we must first resolve the incompatibility between backpropagation, which uses continuous-output neurons and synaptic weights, and neuromorphic designs, which employ spiking neurons and discrete synapses. Our approach is to treat spikes and discrete synapses as continuous probabilities, which allows training the network using standard backpropagation. The trained network naturally maps to neuromorphic hardware by sampling the probabilities to create one or more networks, which are merged using ensemble averaging. To demonstrate, we trained a sparsely connected network that runs on the TrueNorth chip using the MNIST dataset. With a high performance network (ensemble of 64), we achieve 99.42% accuracy at 108 µJ per image, and with a high efficiency network (ensemble of 1) we achieve 92.7% accuracy at 0.268 µJ per image.
Implementing neural models in silicon
, 2004
"... Neural models are used in both computational neuroscience and in pattern recognition. The aim of the first is understanding of real neural systems, and of the second is gaining better, possibly brain-like performance for systems being built. In both cases, the highly parallel nature of the neural sy ..."
Abstract
-
Cited by 1 (0 self)
- Add to MetaCart
(Show Context)
Neural models are used in both computational neuroscience and in pattern recognition. The aim of the first is understanding of real neural systems, and of the second is gaining better, possibly brain-like performance for systems being built. In both cases, the highly parallel nature of the neural system contrasts with the sequential nature of computer systems, resulting in slow and complex simulation software. More direct implementation in hardware (whether digital or analogue) holds out the promise of faster emulation both because hardware implementation is inherently faster than software, and because the operation is much more parallel. There are costs to this: modifying the system (for example to test out variants of the system) is much harder when a full application specific integrated circuit has been built. Fast emulation can permit direct incorporation of a neural model into a system, permitting realtime input and output. Appropriate selection of implementation technology can help to make interfacing the system to external devices simpler. We review the technologies involved, and discuss some example systems. 1 Why implement neural models in silicon? There are two primary reasons for implementing neural models: one is to attempt to gain better, and possibly
A Mixed Analog-Digital Artificial Neural Network Architecture with On-Chip Learning
, 1998
"... This paper presents a novel artificial neural network architecture with on-chip learning capability. The issue of straightforward design-flow integration of an autonomous unit is addressed with a mixed analog-digital approach, by implementing a charge-based artificial neural network which interacts ..."
Abstract
-
Cited by 1 (1 self)
- Add to MetaCart
This paper presents a novel artificial neural network architecture with on-chip learning capability. The issue of straightforward design-flow integration of an autonomous unit is addressed with a mixed analog-digital approach, by implementing a charge-based artificial neural network which interacts with digital control and processing units. We demonstrate the circuit architecture and designflow approach for the case of a Hamming network performing pixel-pattern recognition. Keywords--- Charge-based ANN, mixed-mode ANN hardware architecture, ANN integration design-flow. I. Introduction T HE ABILITY of artificial neural networks (ANN) to acquire knowledge of their surrounding environment and adapt to it, as well as their use of a high degree of computing parallelism makes them very efficient in many application fields including process and quality control, consumer products, optical character and speech recognition, and complex forecasting tasks among many others [1]. Silicon implemen...
Mean Field Bayes Backpropagation: scalable training of multilayer neural networks with binary weights
, 2013
"... Significant success has been reported recently ucsing deep neural networks for classification. Such large networks can be computationally intensive, even after training is over. Implementing these trained networks in hardware chips with a limited precision of synaptic weights may improve their speed ..."
Abstract
-
Cited by 1 (0 self)
- Add to MetaCart
(Show Context)
Significant success has been reported recently ucsing deep neural networks for classification. Such large networks can be computationally intensive, even after training is over. Implementing these trained networks in hardware chips with a limited precision of synaptic weights may improve their speed and energy efficiency by several orders of magnitude, thus enabling their integration into small and low-power electronic devices. With this motivation, we develop a computationally efficient learning algorithm for multilayer neural networks with binary weights, assuming all the hidden neurons have a fan-out of one. This algorithm, derived within a Bayesian probabilistic online setting, is shown to work well for both synthetic and real-world problems, performing comparably to algorithms with real-valued weights, while retaining computational tractability. 1
EURASIP Journal on Applied Signal Processing 2005:7, 993–1004 c ○ 2005 Hindawi Publishing Corporation Object Recognition System-on-Chip Using the Support Vector Machines
, 2004
"... The first aim of this work is to propose the design of a system-on-chip (SoC) platform dedicated to digital image and signal processing, which is tuned to implement efficiently multiply-and-accumulate (MAC) vector/matrix operations. The second aim of this work is to implement a recent promising neur ..."
Abstract
- Add to MetaCart
The first aim of this work is to propose the design of a system-on-chip (SoC) platform dedicated to digital image and signal processing, which is tuned to implement efficiently multiply-and-accumulate (MAC) vector/matrix operations. The second aim of this work is to implement a recent promising neural network method, namely, the support vector machine (SVM) used for real-time object recognition, in order to build a vision machine. With such a reconfigurable and programmable SoC platform, it is possible to implement any SVM function dedicated to any object recognition problem. The final aim is to obtain an automatic reconfiguration of the SoC platform, based on the results of the learning phase on an objects ’ database, which makes it possible to recognize practically any object without manual programming. Recognition can be of any kind that is from image to signal data. Such a system is a general-purpose automatic classifier. Many applications can be considered as a classification problem, but are usually treated specifically in order to optimize the cost of the implemented solution. The cost of our approach is more important than a dedicated one, but in a near future, hundreds of millions of gates will be common and affordable compared to
