Discrete Data Representation for Neural Network Forward Operations
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing methods for performing forward operations in multilayer artificial neural networks are resource-intensive due to the use of successive data representation, which requires complex components and high storage resources, especially for large quantities of high-accuracy weights.
Innovation Solution
A device and method utilizing discrete data representation, which replaces successive data with specified discrete data, allowing operations like multiplication to be performed through bitwise operations, reducing the need for multiplier components and optimizing storage and energy consumption, by employing an instruction caching unit, controller unit, data access unit, interconnecting module, primary operating module, and secondary operating modules that support discrete data representation.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Measurement precision
If successive data representation is used, then calculation accuracy is maintained, but storage resources and device complexity increase significantly
Solution Approach 1:
The patent changes the data representation parameter from successive data (floating-point or fixed-point) to discrete data (quantized values with limited precision). This parameter change allows the system to use simpler components while maintaining acceptable calculation accuracy for neural network forward operations, directly resolving the contradiction between accuracy and complexity.
2Measurement precision
If successive data representation is used, then calculation accuracy is maintained, but storage resources increase
Solution Approach 1:
The patent applies parameter changes by quantizing data into discrete representations (e.g., 1-bit, 2-bit, or 4-bit quantization levels). This reduces the storage requirement from 32 bits per floating-point number to significantly fewer bits, while maintaining sufficient precision for neural network computations, thus resolving the storage resource contradiction.
3Measurement precision
If successive data representation is used, then calculation accuracy is maintained, but the number of multiplier components increases
Solution Approach 1:
The patent substitutes the mechanical multiplication operation with bitwise operations (XOR, AND, shift operations) when using discrete data representation. This substitution eliminates the need for complex multiplier components, reducing device complexity while maintaining calculation accuracy through the mathematical properties of discrete data operations.
4Device complexity
If discrete data representation is used, then storage resources and device complexity are reduced, but operation speed may be affected
Solution Approach 1:
The patent replaces complex multiplication operations with simpler bitwise operations (XOR, AND, shift) that can be executed faster in hardware. This substitution not only reduces device complexity but also improves operation speed, as bitwise operations are inherently faster than multiplication in digital logic circuits.
Data Source
Figure 1
Figure 2~3
Figure 4~6
AI summary
The present disclosure provides a device configured to perform a forward operation of an artificial neural network supporting discrete data representation. The device includes an instruction caching unit, a controller unit, a data access unit, an interconnecting module, a primary operating module, a plurality of secondary operating modules, a discrete data operating module, and a converting module. The forward operation of the multilayer artificial neural network supporting the discrete data representation may be achieved by means of using the device. The data, such as weights and neurons, in the forward operation may be discretely represented. For example, -1, -1/2, 0, 1/2, 1 are not the successive data. A module supporting discrete data operations is provided. Basic operations of the successive data, for example a multiplication operation and an addition operation are replaced with different bitwise operations, for example an exclusive-OR operation and a NOT operation, according to the values of the discrete data. A module configured to convert the successive data into the discrete data is provided. A method for using the device to support a batch normalization operation is provided.