Binary Neural Network Training Using Error-Correcting Codes
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Current deep neural networks (DNNs) face challenges in resource-constrained devices due to high memory and energy consumption, and existing methods for training low-precision binary neural networks are inefficient, requiring floating-point arithmetic and being unsuitable for scalable implementation on devices like smartphones.
Innovation Solution
A data processing system that trains binary neural networks using an error-correcting function to identify and update binary weights within the binary field, eliminating the need for higher arithmetic precision and enabling efficient training on AI chipsets with minimal accuracy loss.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Use of energy by moving object
If binary neural networks are used to reduce memory usage and energy consumption, then resource efficiency is improved, but training efficiency deteriorates due to the need for floating-point arithmetic
Solution Approach 1:
The patent replaces floating-point arithmetic operations with binary field operations (GF(2)) for training binary neural networks. Specifically, gradient computations and weight updates that traditionally require floating-point arithmetic are transformed into binary operations such as XOR and AND operations in the binary field, enabling training to be performed directly in binary without converting to higher precision formats.
Solution Approach 2:
The patent changes the arithmetic precision parameter from floating-point to binary field representation. By formulating the training problem in terms of binary field parameters and using binary field algebra for computations, the system maintains training capability while operating exclusively in binary, thus improving energy efficiency without sacrificing training effectiveness.
2Quantity of substance
If binary neural networks are used to reduce memory usage, then memory consumption is reduced, but training complexity increases due to lack of gradient information
Solution Approach 1:
The patent substitutes the continuous gradient descent mechanism with a binary field-based optimization approach. Instead of computing gradients in floating-point arithmetic, the system uses binary field operations to compute weight updates, replacing the mechanical gradient computation process with algebraic operations in GF(2) that are naturally suited to binary representation.
Solution Approach 2:
The patent introduces binary field algebra as an intermediary framework that bridges the gap between binary network operations and training requirements. This intermediary mathematical structure provides the necessary computational tools for gradient-like operations in the binary domain, simplifying the training process while maintaining memory efficiency.
3Use of energy by moving object
If low-precision arithmetic is used to improve energy efficiency, then energy consumption is reduced, but accuracy deteriorates due to quantization errors
Solution Approach 1:
The patent changes the precision parameter from floating-point to binary field representation and simultaneously changes the computational framework to match. By formulating training objectives, loss functions, and optimization algorithms in terms of binary field operations, the system maintains numerical precision appropriate for binary arithmetic while achieving the energy efficiency benefits of low-precision computation.
Solution Approach 2:
The patent replaces floating-point arithmetic operations with binary field operations throughout the training process. This substitution ensures that all computations are performed in the binary domain using operations native to GF(2), eliminating quantization errors that would arise from truncating floating-point values and maintaining accuracy while reducing energy consumption.
Data Source
Figure 1~3
Figure 4~5
Figure 6(a)~7(c)
AI summary
A data processing system having a binary neural network architecture for receiving a binary network input and in dependence on the network input propagating signals via a plurality of binary processing nodes, in accordance with respective binary weights, to form a network output, the data processing system being configured to train each node of the plurality of binary processing nodes by implementing the node function as an error correcting code (e.g. an r-th-order Reed-Muller code such as the 1-st order Reed-Muller code or cosets of the lst-order Reed-Muller code) function to identify a set of binary weights by channel decoding (e.g. fast Walsh-Hadamard transform algorithm) which minimize, for a given input to that node, any error between the node's output when formed in accordance with the node's current binary weights and a preferred output from the node and to update the weights of that node to be the identified set of binary weights. This training is performed without storing and/or using any higher arithmetic precision weights or other components.