Multi-Layer Neural Network Using Precision Segmentation
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Deep neural networks require significant computational resources due to high precision weights and activations, making them unsuitable for resource-constrained environments, and replacing layers with ultra-low precision layers can reduce task performance.
Innovation Solution
A neural network architecture that combines ultra-low precision layers with higher-precision layers using skip connections and dimension reduction techniques, such as global pooling, to maintain performance while reducing resource usage.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Measurement precision
If higher precision weights and activations are used in deep neural networks, then classification accuracy is improved, but memory and computational resources are significantly increased
Solution Approach 1:
The neural network is segmented into multiple layers with different precision requirements. Early layers use ultra-low precision (1-bit or 2-bit) weights and activations, while later layers use higher precision (32-bit or 16-bit). This segmentation allows the network to achieve high classification accuracy while significantly reducing the memory resources required for storing and processing weights and activations in the majority of layers.
Solution Approach 2:
Different parts of the neural network are assigned different precision levels based on their specific functional requirements. The invention applies ultra-low precision to early layers where it suffices, and higher precision to later layers where it is needed for final classification accuracy. This local differentiation of precision quality optimizes the balance between resource usage and performance.
2Measurement precision
If higher precision weights and activations are used in deep neural networks, then classification accuracy is improved, but computational power requirements are significantly increased
Solution Approach 1:
The computational workload is segmented across layers with different precision requirements. By performing the majority of computations in early layers using ultra-low precision (1-bit or 2-bit), the invention dramatically reduces the computational power required for the bulk of processing. Higher precision computations are reserved only for later layers where they are necessary to maintain classification accuracy.
Solution Approach 2:
The invention changes the precision parameter of weights and activations from uniform high precision to variable precision across layers. Specifically, early layers use ultra-low precision parameters (1-bit or 2-bit) while later layers use higher precision parameters (32-bit or 16-bit). This parameter change optimizes the trade-off between computational power consumption and classification accuracy.
3Quantity of substance
If ultra-low precision layers are used to reduce resource usage, then memory and computational requirements are reduced, but task performance deteriorates
Solution Approach 1:
The network is segmented into ultra-low precision early layers and higher precision later layers. This segmentation ensures that task performance is maintained by using higher precision in the final layers where it directly impacts classification accuracy, while ultra-low precision layers provide sufficient feature extraction with reduced resource requirements.
Solution Approach 2:
Different precision levels are applied locally to different layers based on their functional needs. The invention uses ultra-low precision locally in early layers where feature extraction can be effectively performed with limited precision, and higher precision locally in later layers where classification accuracy is critical. This local quality differentiation maintains overall task performance while reducing total resource usage.
Data Source
AI summary
Provided is multi-layer neural network technique that includes: calculating, from an input and using a first one or more layers of a plurality of layers of a neural network, a first intermediate output; reducing a size of one or more dimensions of the first intermediate output; calculating, from the first intermediate output and using a second one or more layers of the neural network, a second intermediate output (the second one or more layers including one or more ultra-low precision layers); reducing a size of one or more dimensions of the second intermediate output; combining a plurality of reduced intermediate outputs (including the reduced first intermediate output and the reduced second intermediate output) to derive a combined intermediate output; and calculating, using the combined intermediate output and one or more higher-precision layers of the plurality of layers, a neural network output.