Binary Deep Neural Network Training Without Gradient Processing
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing methods for training binary deep neural networks rely on floating-point training, which does not effectively address memory and computational complexity issues and adds complexity to the training process.
Innovation Solution
A device and method for training binary deep neural networks that generate a training signal based on the error between the output of a prototype binary deep neural network and an expected output, allowing for direct optimization of binary weights in the binary domain without gradient processing.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Ease of manufacture
If floating-point training is used for binary deep neural networks, then the model can be trained with existing gradient-descent methods, but the memory and computational complexity requirements remain high
Solution Approach 1:
The patent changes the parameter representation from floating-point to binary domain. Instead of using floating-point parameters during training, the invention uses binary parameters (0 or 1) throughout the training process, fundamentally altering the numerical domain and enabling efficient binary operations that reduce memory and computational requirements while maintaining trainability through specialized binary optimization algorithms
Solution Approach 2:
The patent replaces the gradient-descent optimization mechanism (which relies on floating-point arithmetic and gradient signals) with a binary optimization mechanism. This substitution eliminates the need for gradient computation and floating-point multiplication, using instead binary-compatible optimization approaches that operate directly in the binary domain, thereby reducing computational complexity and memory usage
2Quantity of substance
If binary parameters are used in the deep neural network, then the model footprint and training memory are reduced, but the parameters become discrete and cannot be optimized with gradient-descent
Solution Approach 1:
The patent replaces the gradient-descent optimization mechanism (which relies on floating-point arithmetic and gradient signals) with a binary optimization mechanism. This substitution eliminates the need for gradient computation and floating-point multiplication, using instead binary-compatible optimization approaches that operate directly in the binary domain, thereby reducing computational complexity and memory usage
Solution Approach 2:
The patent changes the parameter representation from floating-point to binary domain. Instead of using floating-point parameters during training, the invention uses binary parameters (0 or 1) throughout the training process, fundamentally altering the numerical domain and enabling efficient binary operations that reduce memory and computational requirements while maintaining trainability through specialized binary optimization algorithms
3Quantity of substance
If binarization is applied during training, then binary parameters are achieved, but the training process complexity increases and memory issues are not solved
Solution Approach 1:
The patent extracts the binarization step from the training process and applies it continuously throughout training rather than as a separate post-processing step. By maintaining binary parameters throughout the entire training pipeline and removing floating-point operations, the invention simplifies the training process architecture and eliminates the complexity of managing dual-precision systems, thereby reducing training process complexity while achieving binary parameters
Data Source
AI summary
A device for training a binary deep neural network, where the device includes a processor, is configured to: generate a training signal in dependence on an error between an output of a prototype version of the binary deep neural network and an expected output, the prototype version of the binary deep neural network having multiple binary weights each having a respective value; and in dependence on the training signal, output for each binary weight of the prototype version of the binary deep neural network a respective decision to invert or maintain the respective value of the respective binary weight. This may allow the device to train a deep neural network including binary parameters directly in the binary domain without the need for gradient processing methods.