Inline Compression Metadata Using Hash Blocks in Data Buffers
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing data compression techniques require separate metadata buffers to indicate compressed and uncompressed data blocks, leading to inefficiencies in storage and bandwidth usage, and are prone to collisions when hash values match uncompressed data.
Innovation Solution
The use of inline hash blocks within a buffer to indicate compression status, where blocks are grouped and hash functions are chosen to prevent collisions, allowing for efficient storage and transmission of compressed data without separate metadata buffers.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If separate metadata buffers are used to indicate compressed and uncompressed data blocks, then compression status can be tracked, but storage and bandwidth requirements increase
Solution Approach 1:
The patent combines the metadata buffer and data buffer into a single unified buffer structure. Hash blocks containing compression status information are interleaved directly with data blocks within the same buffer memory, eliminating the need for separate metadata storage and reducing overall memory requirements.
Solution Approach 2:
The metadata (hash blocks) are nested within the data buffer structure itself. Each hash block is embedded alongside corresponding data blocks, creating a hierarchical organization where compression metadata is contained within the same memory space as the actual data, optimizing memory utilization.
2Productivity
If hash functions are used to indicate compressed blocks, then compression metadata can be generated, but collisions occur when hash values match uncompressed data
Solution Approach 1:
The buffer is segmented into fixed-size blocks where every fourth block is designated as a hash block and the remaining three blocks are data blocks. This segmentation ensures that hash values are stored in dedicated locations separate from uncompressed data blocks, preventing hash values from being misinterpreted as uncompressed data.
Solution Approach 2:
The hash function acts as an intermediary that transforms compressed data blocks into unique hash values stored in hash blocks. These hash values serve as indicators that mediate between the compressed data and the system needing to identify compression status, with the segmented buffer structure ensuring unambiguous interpretation.
3Quantity of substance
If the same memory allocation is used for compressed and uncompressed data, then memory efficiency is improved, but additional mechanisms are needed to manage compression metadata
Solution Approach 1:
The unified buffer serves multiple functions simultaneously: it stores both compressed and uncompressed data blocks, contains hash blocks for compression status indication, and provides a structured format for both storage and retrieval operations. This multi-functionality eliminates the need for separate metadata management mechanisms while maintaining memory efficiency.
Data Source
AI summary
Techniques for handling data compression in which metadata that indicates which portions of data are compressed are which portions of data are not compressed are disclosed. Segments of a buffer referred to as block groups store compressed blocks of data along with uncompressed blocks of data and hash blocks. If a block group includes a block that is a hash of another block in the block group, then the other block is considered to be compressed. If the block group does not include a block that is a hash of another block in the block group, then the blocks in the block group are uncompressed. The hash function to generate the hash is selected to prevent “collisions,” which occur when the data being stored in the buffer is such that it is possible for a hash block and an uncompressed block to be the same.