Deep Learning Operator Optimization via L1 Cache Secondary Quantization
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing deep learning neural networks on terminal devices face high computational complexity and resource consumption due to large model parameters, leading to slow quantization speed and increased power consumption when using fixed-point quantization methods like the Buffer method, which reads data from the L2 Cache.
Innovation Solution
The method optimizes deep learning operators by reading target data from the L1 cache of an image processor, performing secondary quantization operations, and writing results into the main memory, leveraging the faster access speeds of the L1 cache to improve neural network quantization efficiency and reduce resource occupation and power consumption.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Productivity
If the Buffer method is used to read data from L2 Cache for fixed-point quantization, then quantization can be performed, but the quantization speed is slow and power consumption is high
Solution Approach 1:
The patent pre-loads quantization data from L2 cache to L1 cache before the actual quantization operation. This preliminary action stores frequently accessed data in the faster L1 cache, reducing the need for slow L2 cache accesses during quantization execution, thereby improving both speed and reducing power consumption.
Solution Approach 2:
The patent introduces L1 cache as an intermediary between L2 cache and the quantization processing unit. Data flows from L2 cache to L1 cache first, then to the processing unit. This intermediary layer buffers data transfers, reducing direct L2 cache accesses and improving overall system efficiency.
2Measurement precision
If larger scale model parameters are used to achieve better task performance, then accuracy improves, but computational complexity and resource consumption increase
Solution Approach 1:
The patent applies fixed-point quantization to change the numerical precision parameters of the neural network model. By converting floating-point numbers to fixed-point representations with lower precision, the model maintains adequate performance while significantly reducing computational complexity and resource requirements for deployment on terminal devices.
Solution Approach 2:
The patent uses lower-precision fixed-point data types instead of high-precision floating-point types. This substitution of cheaper, lower-precision representations allows the model to run efficiently on resource-constrained terminal devices while maintaining sufficient accuracy for practical applications.
3Speed
If data is read from L2 Cache for quantization operations, then quantization can be performed, but access speed is slow
Solution Approach 1:
The patent pre-loads quantization data from L2 cache to L1 cache before the actual quantization operation. This preliminary action stores frequently accessed data in the faster L1 cache, reducing the need for slow L2 cache accesses during quantization execution, thereby improving both speed and reducing power consumption.
Solution Approach 2:
The patent introduces a new memory access dimension by utilizing L1 cache as an intermediate storage layer. Instead of direct L2 cache access, data is first transferred to L1 cache, creating a two-stage memory access hierarchy that exploits the speed difference between L1 and L2 caches to improve overall access performance.
Data Source
Figure 1
Figure 2
Figure 3~5
AI summary
A method for optimizing a deep learning operator, includes: calling (S101, S203) a method of reading an image object to read target data from an L1 cache of an image processor to the processor in response to detecting the target data in the L1 cache, performing (S102, S204) a secondary quantization operation on the target data in the processor to obtain an operation result and writing (S103, S205) the operation result into a main memory of the image processor. The target data is fixed-point data obtained after performing a quantization operation on data to be quantized in advance and the data to be quantized is one of the following: float-point data of an initial network layer of the neural network model and fixed-point data outputted from a network layer previous to the current network layer.