Binary Neural Network Training Using Error-Correcting Binary Weights

Resolve Bottlenecks,
Find Innovative SolutionsGenerate Solutions

Solution Overview

Problem

Current deep neural networks (DNNs) face challenges in resource-constrained devices due to high memory and energy consumption, and existing methods for training low-precision binary neural networks are inefficient, requiring floating-point arithmetic and being unsuitable for scalable implementation on smartphones.

Innovation Solution

A data processing system that trains binary neurons and neural networks using binary field arithmetic, implementing an error correcting function to identify and update binary weights without higher arithmetic precision, allowing for efficient training and inference on AI chipsets.

Engineering Contradictions & Design Principles

VSEngineering Contradiction Analysis

1Use of energy by moving object

If binary neural networks are used to reduce memory and energy consumption, then resource efficiency is improved, but training efficiency deteriorates due to requiring floating-point arithmetic

Engineering Contradiction:

Improveenergy consumptionVSAvoidtraining efficiency

Core Design Contradiction:

Use of energy by moving objectVSProductivity

Solution Approach 1:

The patent replaces floating-point arithmetic operations with binary field arithmetic operations during training. Specifically, it uses binary addition, subtraction, and multiplication in the binary field GF(2) instead of conventional floating-point arithmetic, which enables training to be performed using simple binary operations that can be efficiently implemented in hardware without requiring high-precision arithmetic units.

Inventive Principle:

Principle #28Mechanics substitution (Replace mechanical system)

2Device complexity

If binary field arithmetic is used for training, then device complexity is reduced, but measurement precision deteriorates in weight representation

Engineering Contradiction:

Improvearithmetic precision requirementVSAvoidweight precision

Core Design Contradiction:

Device complexityVSMeasurement precision

Solution Approach 1:

The patent changes the numerical parameter system from floating-point arithmetic to binary field arithmetic (GF(2))). In this binary field, weights and activations are represented as binary values {0, 1} with arithmetic operations defined modulo 2. This parameter change reduces the precision requirements while maintaining the ability to represent and differentiate weight relationships through the binary optimization process.

Inventive Principle:

Principle #35Parameter changes

Data Source

PatentUS20220237456A1Design and Training of Binary Neurons and Binary Neural Networks with Error Correcting Codes

Publication Date: 2022.07.28 HUAWEI TECH CO LTD

AI summary

A data processing system having a neural network architecture for receiving a binary network input and, in dependence on the binary network input, propagating signals via a plurality of processing nodes, in accordance with respective binary weights, to form a network output, the data processing system being configured to train a node by implementing an error correcting function to identify a set of binary weights which minimize, for a given input to the node, any error between an output of the node when formed in accordance with current binary weights of the node and a preferred output from the node and to update the binary weights of the node to be the identified weights. This training is performed without storing and/or using any higher arithmetic precision weights or other components.