Binary Neural Network Training via Shadow Weight Distributions
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Training binary neural networks is inefficient due to the need for frequent rounding steps during forward propagation, leading to noise and potential degradation in task-performance, especially when switching from full-precision networks, which often requires widening layers and increased hardware resources.
Innovation Solution
A training method that uses distributions of weights and activations instead of point estimates, allowing for gradient descent on probabilistic parameters, reducing rounding noise and enabling efficient implementation on low-resource hardware by sampling from Bernoulli-like and Gaussian distributions, and storing uncertainty information for improved accuracy and resource efficiency.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Quantity of substance
If binary neural networks use frequent rounding steps during forward propagation, then hardware resource consumption is reduced, but training stability deteriorates and task-performance degrades
Solution Approach 1:
The patent applies preliminary action by performing rounding operations during the backward propagation phase rather than during forward propagation. The method computes gradients with respect to continuous shadow weights first, then applies rounding to obtain binary weight updates. This preliminary computation with continuous values before final binary conversion stabilizes training while still achieving hardware efficiency.
Solution Approach 2:
The patent introduces shadow weights as an intermediary between binary weights and gradient computations. These shadow weights serve as continuous-valued intermediaries that allow stable gradient-based optimization during training, while the actual binary weights remain discrete. The shadow weights mediate between the requirements for hardware efficiency (binary values) and training stability (continuous optimization).
2Quantity of substance
If binary neural networks use frequent rounding steps during forward propagation, then hardware resource consumption is reduced, but task-performance deteriorates
Solution Approach 1:
The patent performs preliminary gradient computations using continuous shadow weights before applying binary rounding. This allows the network to learn optimal weight directions in continuous space, and only then converts to binary values for hardware deployment. The preliminary continuous optimization preserves task-performance while enabling subsequent binary implementation.
Solution Approach 2:
The patent changes the parameter representation from directly using binary weights to using continuous shadow weights during training. This parameter transformation allows the use of standard gradient descent optimization algorithms while maintaining the ability to convert to binary format. The parameter change enables high-performance training that can be transferred to resource-efficient binary deployment.
3Use of energy by moving object
If binary neural networks are implemented without floating-point multiplication units, then energy consumption is reduced, but computational precision is limited
Solution Approach 1:
The patent uses shadow weights as intermediaries that maintain continuous precision during computation while the final weight storage and hardware implementation use binary format. This intermediary approach allows the system to achieve both low energy consumption (through binary hardware implementation) and high computational precision (through continuous shadow weight optimization during training).
Data Source
Figure 1
Figure 2
Figure 3
AI summary
A method (100) for training an artificial neural network (1) that is at least partially implemented as a binary neural network (2), comprising • receiving (110) training input values (11) for the inputs xi of the artificial neural network (1), • processing (120) training input values (11) to obtain output values (12), • applying (130), to obtained output values (12), a loss function (13); • updating (140) training weights wlj and branching back (150) to processing (120) until rating (13a) by the loss function (13) meets a predetermined termination criterion (160); and • configuring (170) actual artificial neural network (1, 2) according to finally obtained training weights wlj, wherein processing (120) comprises: • determining (121), for each neuron I in the layer (21-23), a distribution Al of pre-activations al using a distribution Wlj of weights wlj; and • determining (122) a distribution Hl of output values hl by applying, to distribution Al, predetermined thresholding function g.