Structured Sparse Compression for Faster Neural Network Hardware
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing data compression techniques for sparse data structures in artificial neural networks are inefficient, leading to high bandwidth and memory usage, which hinders the processing and storage of large datasets.
Innovation Solution
A method and system for compressing sparse data structures by generating metadata that identifies significant elements and restructuring the data accordingly, allowing for a structured sparsity format that simplifies hardware processing and reduces data size, using instructions like GENMETADATA and GATHER to select and gather values from the original data structure, resulting in a compressed data structure with a consistent M:N compression ratio.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Quantity of substance
If conventional compression techniques are used for sparse data structures, then data size is reduced, but processing efficiency decreases and hardware complexity increases
Solution Approach 1:
The patent segments the data structure into significant elements and insignificant elements, storing only the significant ones in a compressed format while maintaining their positional relationships through metadata. This segmentation allows the data to be compressed without requiring complex processing during operations, as the hardware can efficiently navigate the structured sparse format using the metadata pointers.
Solution Approach 2:
The patent introduces metadata as an intermediary structure that bridges the compressed data and the original data structure. The metadata contains positional information that enables hardware to efficiently access and process the compressed data without requiring complex decompression logic, thus maintaining processing efficiency while achieving compression.
2Quantity of substance
If conventional compression techniques are used for sparse data structures, then data size is reduced, but memory usage increases
Solution Approach 1:
The patent segments the data structure to identify and store only significant elements, reducing the overall data volume. By dividing the data into compressed portions and metadata portions, the system optimizes memory utilization by storing only necessary information in a structured format that minimizes memory footprint.
Solution Approach 2:
The patent changes the representation parameters of the data structure by transitioning from a dense format to a structured sparse format with M:N compression ratio. This parameter change involves storing data as non-zero elements with their positional metadata, fundamentally altering how data occupies memory space and reducing overall memory usage.
3Productivity
If structured sparsity format is implemented, then processing efficiency is enhanced, but device complexity increases
Solution Approach 1:
The patent applies local quality by creating a structured sparse format where different portions of the data structure have different properties. The significant elements are stored with their positional metadata in a organized manner, allowing hardware to efficiently process only the relevant portions of data while maintaining simple access patterns through the metadata structure.
Solution Approach 2:
The patent performs preliminary organization of data into a structured sparse format during data preparation, arranging elements and their metadata in advance. This preliminary action enables hardware to process the data efficiently without requiring complex runtime processing logic, as the structure is already optimized for fast access and computation.
Data Source
AI summary
In artificial neural networks, and other similar applications, there is typically a large amount of data involved that is considered sparse data. Due to the large size of the data involved in such applications, it is helpful to compress the data to save bandwidth resources when transmitting the data and save memory resources when storing the data. Introduced herein is a compression technique that selects elements with significant values from data and restructures them into a structured sparse format. By generating metadata that enforces the structured sparse format and organizing the data according to the metadata, the introduced technique not only reduces the size of the data but also consistently places the data in a particular format. As such, hardware can be simplified and optimized to process the data much faster and much more efficiently than the conventional compression techniques that rely on a non-structured sparsity format.