ML Accelerator Compressed Data Loading Without Zero-Value Stalls
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Machine learning accelerators face inefficiencies in data loading due to the sparsity of weights and activations in deep neural networks, leading to energy-intensive loading of zero-valued data that does not contribute to computations, and existing solutions like Zero Value Compute schemes are limited by data loading rates that do not keep compute units busy all the time.
Innovation Solution
Implementing a data controller with a compressed local data re-user and a zero compute load skipper that selectively loads only useful, non-zero data sections into the processor engine, utilizing sparsity bitmaps and pointer values to optimize data transfer and reduce energy consumption by minimizing unnecessary data loading.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Productivity
If data is loaded at high rate to keep compute units busy, then productivity is improved, but energy consumption increases due to loading zero-valued data
Solution Approach 1:
The patent extracts and removes zero-valued data sections from the data stream before loading into the processor engine. The data controller identifies consecutive zero-valued sections using sparsity bitmaps and excludes them from loading, thereby reducing energy consumption while maintaining high compute unit utilization by continuously loading non-zero data sections.
Solution Approach 2:
The system uses sparsity bitmaps that are generated as part of the data compression process to automatically identify and skip zero-valued sections. The data controller autonomously makes loading decisions based on these bitmaps without requiring external intervention, enabling energy-efficient operation while maintaining productivity.
2Reliability
If all data sections are loaded into processor engine, then completeness of data processing is improved, but energy consumption and memory access increase
Solution Approach 1:
The patent extracts only the necessary non-zero data sections from the compressed data stream. The data controller uses sparsity bitmaps to identify which sections contain non-zero values and loads only those sections into the processor engine, ensuring complete processing of all meaningful data while eliminating energy-wasting loads of zero-valued sections.
Solution Approach 2:
The sparsity bitmaps are generated in advance during data compression, providing a roadmap of which data sections contain non-zero values. This preliminary information enables the data controller to pre-identify and selectively load only the necessary data sections, ensuring processing completeness while minimizing energy consumption.
3Use of energy by moving object
If compressed data is used to reduce memory access, then energy consumption is reduced, but data loading rate decreases
Solution Approach 1:
The patent extracts non-zero data sections from the compressed data stream using sparsity bitmaps. By removing zero-valued sections before loading, the system reduces the actual amount of data that needs to be transferred and processed, thereby maintaining high data loading rates while preserving the energy benefits of compression.
Solution Approach 2:
The data controller dynamically adjusts the loading rate based on the sparsity characteristics of the data. When zero-valued sections are identified, the controller pauses loading and skips directly to non-zero sections, creating a dynamic loading pattern that maintains high effective data loading rates while minimizing energy consumption.
Data Source
Figure 1
Figure 2
Figure 3
AI summary
Methods, apparatus, systems, and articles of manufacture to load data into an accelerator are disclosed. An example apparatus includes data provider circuitry to load a first section and an additional amount of compressed machine learning parameter data into a processor engine. Processor engine circuitry executes a machine learning operation using the first section of compressed machine learning parameter data. A compressed local data re-user circuitry determines if a second section is present in the additional amount of compressed machine learning parameter data. The processor engine circuitry executes a machine learning operation using the second section when the second section is present in the additional amount of compressed machine learning parameter data.