Runtime Compression Format Conversion for Sparse DNN Memory Bandwidth
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Deep neural networks (DNNs) face performance bottlenecks due to limited memory bandwidth, and using a single fixed compression format prior to runtime can result in suboptimal compression formats for unpredictable future workloads.
Innovation Solution
A runtime data-format optimizer for processing elements, incorporating a sparsity-detection circuit and a compression-conversion circuit, dynamically selects optimal compression formats based on detected sparsity patterns and performance models that consider latency, energy consumption, and conversion costs.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Device complexity
If a fixed compression format is selected before deployment, then device complexity is reduced, but adaptability to different sparsity patterns deteriorates
Solution Approach 1:
The patent implements a runtime reconfigurable compression converter that dynamically selects and switches between different compression formats (CSR, CSC, COO, etc.) based on the actual sparsity patterns of incoming data. This dynamic adaptation allows the system to optimize compression performance for each specific workload without requiring complex pre-deployment analysis, resolving the contradiction between device complexity and adaptability.
2Ease of operation
If compression format is fixed at design time, then ease of operation is improved, but productivity deteriorates due to suboptimal compression ratios
Solution Approach 1:
The system employs an automated sparsity-detection circuit and performance-model-based selection mechanism that autonomously determines the optimal compression format without requiring manual intervention or complex user configuration. This self-service approach maintains ease of operation while significantly improving productivity by ensuring optimal compression ratios are achieved for each workload through automatic adaptation to sparsity patterns.
3Adaptability or versatility
If runtime compression format conversion is implemented, then adaptability to sparsity patterns is improved, but device complexity increases
Solution Approach 1:
The patent introduces an intermediary performance model that acts as a bridge between the sparsity-detection circuit and the compression-conversion circuit. This performance model pre-evaluates multiple compression formats and provides selection guidance, thereby reducing the complexity burden on the runtime converter while maintaining high adaptability to different sparsity patterns.
4Productivity
If multiple compression formats are supported at runtime, then productivity is improved through optimal compression, but loss of time increases due to format conversion overhead
Solution Approach 1:
The system performs preliminary evaluation of multiple compression formats using a performance model during offline analysis, storing the results for rapid runtime selection. This preliminary action avoids time-consuming format conversions at runtime by pre-determining the optimal format based on sparsity patterns, thereby improving productivity while minimizing time loss through efficient format selection.
Data Source
AI summary
A runtime data-format optimizer for a processing element includes a sparsity-detector and a compression-converter. The sparsity-detector selects a first compression-conversion format during a runtime of the processing element based on a performance model that is based on a first sparsity pattern of first data stored in a first memory that is exterior to the processing element and a second sparsity pattern of second data that is to be stored in a second memory within the processing element. The second sparsity pattern is based on a runtime configuration of the processing element. The first data is stored in the first memory using a first compression format and the second data is to be stored in the second memory using a second compression format. The compression-conversion circuit converts the first compression format of the first data to be the second compression format of the second data based on the first compression-conversion format.