Neural Network Cache Decompression for Bandwidth-Limited Processing
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Neural networks processing in electronic devices face challenges due to the large amount of feature map and weight data that exceeds local storage capacity, necessitating frequent data transfer between memory, which increases bandwidth requirements and processing overhead.
Innovation Solution
Implementing a data processor with a cache system that caches decompressed neural network data and uses compression codecs to decode compressed data only when needed, reducing the number of decompression operations and data transfer.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Speed
If feature map and weight data are stored in local on-chip storage, then processing speed is improved, but storage capacity is insufficient for large neural network data
Solution Approach 1:
The patent implements a hierarchical storage architecture where decompressed data is cached in on-chip storage (L1/L2 caches) while compressed data resides in off-chip memory. This nested structure allows the system to keep frequently accessed decompressed data in fast on-chip storage while maintaining large compressed data sets in off-chip memory, effectively resolving the contradiction between processing speed and storage capacity.
Solution Approach 2:
The system performs preliminary decompression of neural network data and caches the decompressed versions in on-chip storage before they are needed for processing. By anticipating data access patterns and pre-decompressing data, the system ensures that processing units can immediately access decompressed data without waiting for decompression during execution, thereby improving processing speed while managing limited on-chip storage capacity.
2Ease of operation
If data is frequently transferred between memory and processor, then data availability is improved, but bandwidth requirements and processing overhead increase
Solution Approach 1:
The system performs preliminary decompression of neural network data from compressed format and caches the decompressed data in on-chip storage before processing. This advance preparation ensures that processing units can immediately access decompressed data without repeated transfers during execution, reducing bandwidth requirements and processing overhead while maintaining data availability.
Solution Approach 2:
The patent implements a data flow architecture where decompressed data is continuously cached in on-chip storage and made available to processing units without interruption. The system maintains continuous data availability by ensuring decompressed data remains in fast on-chip storage throughout the processing pipeline, eliminating the need for repeated data transfers between off-chip memory and processor, thereby reducing bandwidth consumption and processing overhead.
3Ease of operation
If decompression is performed frequently, then data accessibility is improved, but processing overhead increases
Solution Approach 1:
The system performs preliminary decompression of neural network data and caches the decompressed versions in on-chip storage before they are needed for processing. By decompressing data in advance and storing it in fast on-chip memory, the system ensures that processing units can immediately access decompressed data without triggering decompression operations during execution, thereby improving data accessibility while reducing processing overhead.
Solution Approach 2:
The patent creates copies of decompressed data in on-chip storage from the compressed data in off-chip memory. These cached copies remain in fast on-chip storage throughout processing, allowing multiple processing units to access the same decompressed data without requiring repeated decompression operations. This copying mechanism significantly reduces processing overhead while maintaining high data accessibility.
Data Source
AI summary
A data processor is disclosed that includes a processing unit operable to process neural network data, and a cache system operable to cache neural network data for the processing unit. When neural network data is required for processing, the processing unit issues a request for the neural network data to the cache system, and if the requested data is not cached in the cache system, a compression codec is caused to decode a part of a compressed neural network data stream that encodes the requested neural network data so as to provide the requested neural network data.