Read-Only Data Compression Instructions With Prefetch Buffer Decompression
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Current memory compression techniques either require OS defragmentation, incurring performance overhead, or reduce effective memory bandwidth due to metadata access overhead, and do not effectively address the energy-intensive data transfer between memory and processor in machine learning workloads, particularly for read-only neural network weights.
Innovation Solution
A processor compression instruction that compresses multiple adjacent data blocks of read-only data into one block, storing it in memory and decompressing it in a prefetch buffer for improved bandwidth and latency, using general compression algorithms like Bit-Plane Compression or Base-delta-immediate compression, and storing compressed data in cache to enhance cache bandwidth.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Productivity
If data is compressed into one block and stored in memory, then memory bandwidth is improved, but metadata access overhead increases
Solution Approach 1:
The patent extracts the read-only data (neural network weights) from the general memory space and stores it in a dedicated prefetch buffer. This separation allows the compression metadata to be stored alongside the data without interfering with other memory operations, reducing metadata access overhead while maintaining improved memory bandwidth through compression.
Solution Approach 2:
The patent introduces a prefetch buffer as an intermediary between memory and the processing unit. This buffer serves as a mediator that holds both compressed data and its metadata, allowing the processing unit to access decompressed data without incurring repeated metadata access overhead. The prefetch buffer acts as a local cache that resolves the contradiction by providing fast access to both data and metadata simultaneously.
2Quantity of substance
If compression is used to increase effective memory capacity, then cache capacity is improved, but energy consumption increases due to compression/decompression operations
Solution Approach 1:
The patent performs compression of read-only data in advance before the data is needed for processing. By pre-compressing the neural network weights and storing them in the prefetch buffer with their metadata, the system avoids energy-intensive compression operations during actual inference workloads. The decompression happens automatically from the prefetch buffer, reducing overall energy consumption while maintaining increased effective memory capacity.
3Stability of the object's composition
If OS defragmentation is used to manage compressed data, then data organization is improved, but performance overhead increases
Solution Approach 1:
The patent implements a self-organizing system where the prefetch buffer automatically manages compressed data blocks and their associated metadata without requiring OS defragmentation. The buffer structure inherently maintains proper data organization through its design, with compression metadata stored alongside the compressed data blocks. This self-service approach eliminates performance overhead associated with external defragmentation operations while maintaining stable data organization.
Data Source
AI summary
A processor that includes compression instructions to compress multiple adjacent data blocks of uncompressed read-only data stored in memory into one compressed read-only data block and store the compressed read-only data block in multiple adjacent blocks in the memory is provided. During execution of an application to operate on the read-only data, one of the multiple adjacent blocks storing the compressed read-only block is read from memory, stored in a prefetch buffer and decompressed in the memory controller. In response to a subsequent request during execution of the application for an adjacent data block in the compressed read-only data block, the uncompressed adjacent block is read directly from the prefetch buffer.