Discrete Neural Network Forward Propagation With Bitwise Logic
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Conventional methods for supporting the forward propagation of multilayer artificial neural networks (MNNs) using general-purpose processors and GPUs require significant computational resources and storage space due to the use of continuous data representation, which is structurally complex and inefficient for MNNs with numerous high-precision weight values.
Innovation Solution
An apparatus and method for MNN forward propagation that utilizes a computation module with a master and slave computation module, converting continuous data to discrete values, and employing bit-shifting, bitwise operations, and prestored tables to reduce complexity and resource consumption.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If continuous data representation is used to store floating-point numbers and fixed-point numbers, then the neural network can perform computations, but the computational resources and storage space consumption increase significantly
Solution Approach 1:
The patent changes the data representation parameter from continuous (floating-point/fixed-point) to discrete (one-hot encoding). This transforms the storage requirement from 32 bits per number to 2 bits per number (since 2^2 = 4 possible values), dramatically reducing storage space consumption while maintaining computational capability through discrete math operations
Solution Approach 2:
The patent uses one-hot encoding to represent discrete values, where each value is represented by a binary vector with only one bit set to 1. This copying approach allows the system to work with discrete data using standard binary logic, enabling efficient computation without requiring complex continuous arithmetic hardware
2Adaptability or versatility
If continuous data representation is used, then the neural network can execute general-purpose instructions, but the hardware design becomes structurally more complex
Solution Approach 1:
The patent adopts a simplified discrete data processing approach that uses basic binary logic operations (AND, OR, NOT) instead of complex continuous arithmetic units. These basic logic operations are implemented with simple transistor-level circuits that are cheaper and less complex than floating-point arithmetic units, while still providing sufficient computational power for neural network operations
Solution Approach 2:
The patent divides the computational task into discrete steps using one-hot encoded vectors. Each computation step operates on simplified discrete representations, breaking down the complex continuous computation into manageable discrete operations that can be handled by simpler hardware components
3Device complexity
If discrete data representation is used, then the storage space and hardware complexity are reduced, but the computational operations become more specialized
Solution Approach 1:
The patent designs a universal discrete data processing unit that can handle various neural network operations (addition, multiplication, activation functions) by using the same basic one-hot encoding mechanism and binary logic operations. This multi-functional approach maintains computational flexibility while using simplified hardware
Data Source
AI summary
Aspects for forward propagation of a multilayer neural network (MNN) in a neural network processor are described herein. As an example, the aspects may include a computation module that includes a master computation module and one or more slave computation modules. The master computation module may be configured to receive one or more groups of MNN data. The one or more groups of MNN data may include input data and one or more weight values and wherein at least a portion of the input data and the weight values are stored as discrete values. The one or more slave computation modules may be configured to calculate one or more groups of slave output values based on a data type of each of the one or more groups of MNN data.