Binary Neural Network Training Using Error-Correcting Codes
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Current methods for training low-precision deep neural networks (DNNs) are inefficient and unsuitable for resource-constrained devices like smartphones, as they require high computational resources and cannot be scaled effectively, leading to accuracy losses and energy inefficiencies.
Innovation Solution
A method for training binary neural networks using binary field arithmetic, transforming the training problem into a communication channel decoding problem, utilizing Reed-Muller codes and error-correcting codes to adapt weights without storing higher precision values, enabling efficient training on devices with limited resources.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Use of energy by moving object
If binary neural networks are used to reduce memory usage and energy consumption, then resource efficiency is improved, but training becomes computationally intractable and accuracy deteriorates
Solution Approach 1:
The patent replaces traditional gradient-based optimization mechanics with a statistical physics-based training approach. Instead of using backpropagation and gradient descent, the invention employs a training algorithm that samples binary weight configurations according to a probability distribution derived from training data, making the training process suitable for binary neural networks without requiring complex higher-precision computations.
Solution Approach 2:
The patent changes the parameter representation from continuous floating-point weights to discrete binary weights. By formulating the training problem in terms of binary parameters and using a statistical approach that directly operates on binary configurations, the method enables efficient training of binary neural networks while maintaining accuracy and reducing energy consumption.
2Quantity of substance
If binary neural networks are used to reduce memory usage, then memory consumption is reduced, but training accuracy and model performance deteriorate
Solution Approach 1:
The patent replaces gradient-based optimization with a statistical sampling approach that directly searches the binary weight space. This substitution allows the training process to find accurate binary weight configurations without relying on continuous approximations, thereby maintaining high accuracy while using only binary parameters that reduce memory usage.
Solution Approach 2:
The training algorithm incorporates feedback from training data to iteratively improve binary weight configurations. By sampling weight configurations and evaluating their performance on training examples, the method receives feedback that guides the search toward configurations that maximize accuracy, ensuring high performance despite the limited representation capacity of binary weights.
3Measurement precision
If standard floating-point arithmetic is used for DNN training, then computational accuracy is maintained, but energy consumption and resource usage increase significantly
Solution Approach 1:
The patent changes the arithmetic precision from standard floating-point to binary throughout the entire training process. By formulating both the network parameters and the training computations in binary, the method maintains computational accuracy within the binary domain while achieving significant energy savings compared to floating-point arithmetic.
Solution Approach 2:
The patent substitutes floating-point arithmetic operations with binary arithmetic operations in the training process. Instead of using floating-point multiplications and additions, the method employs binary operations that are natively supported by low-precision hardware, thereby reducing energy consumption while maintaining accuracy through the statistical training approach.
4Ease of operation
If binary neural networks are deployed on resource-constrained devices, then portability and efficiency are improved, but training capability and scalability are limited
Solution Approach 1:
The patent replaces resource-intensive gradient-based training mechanics with a statistically efficient training approach that is suitable for binary networks. This substitution reduces the computational burden of training, enabling binary neural networks to be trained and scaled on resource-constrained devices while maintaining portability and efficiency.
Solution Approach 2:
By changing all parameters to binary and using a training algorithm that operates efficiently in the binary domain, the patent enables training of scalable binary neural networks on devices with limited resources. The binary parameter representation reduces memory requirements and computational complexity, improving both portability and training scalability.
Data Source
Figure 1~3
Figure 4~5
Figure 6(a)~7(c)
AI summary
A data processing system having a neural network architecture for receiving a binary network input and, in dependence on the binary network input, propagating signals via a plurality of processing nodes, in accordance with respective binary weights, to form a network output, the data processing system being configured to train a node by implementing an error correcting function to identify a set of binary weights which minimize, for a given input to the node, any error between an output of the node when formed in accordance with current binary weights of the node and a preferred output from the node and to update the binary weights of the node to be the identified weights. This training is performed without storing and/or using any higher arithmetic precision weights or other components.