Compressed Data Management in Fixed-Size Block Storage
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Managing compressed data in fixed-size block storage systems is challenging due to compressed buffers of arbitrary size not fitting neatly into fixed-size physical locations, and combining compression with deduplication increases complexity, while compressing small input blocks is sub-optimal and negatively affects deduplication ratios.
Innovation Solution
A storage system that dynamically manages metadata and reference tracking for compressed data, allowing for flexible block sizing based on compressibility, using techniques like fingerprinting and reference counting to optimize storage and deduplication efficiency.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Quantity of substance
If input data blocks are compressed into smaller physical blocks, then storage efficiency is improved, but the complexity of metadata management and reference tracking increases
Solution Approach 1:
The patent implements nested metadata structures where compressed buffer metadata contains references to input block metadata, which in turn contains references to physical block metadata. This hierarchical nesting allows the system to manage compressed data efficiently by organizing metadata in a structured manner that reduces overall management complexity despite the increased number of metadata objects.
Solution Approach 2:
The patent introduces compressed buffer metadata as an intermediary layer between input block metadata and physical block metadata. This intermediary structure facilitates reference tracking and garbage collection by providing a centralized point to manage references from multiple input blocks to compressed buffers, thereby reducing the complexity of direct many-to-many reference management.
2Device complexity
If input block size is increased to reduce metadata overhead, then metadata overhead is reduced, but deduplication ratio deteriorates
Solution Approach 1:
The patent segments the deduplication process into two stages: first, input blocks are hashed and deduplicated individually using their own fingerprints; second, compressed buffers are deduplicated using fingerprints generated from compressed data. This segmentation allows small input blocks to maintain good deduplication ratios while the overall metadata overhead is managed through the compressed buffer level.
Solution Approach 2:
The patent implements dynamic block sizing where the effective deduplication block size varies based on compressibility. Highly compressible data effectively creates larger deduplication units, while less compressible data maintains smaller units. This dynamic adjustment optimizes both deduplication ratio and metadata overhead without requiring fixed large block sizes.
3Quantity of substance
If small input blocks are compressed, then storage space is reduced, but compression efficiency decreases
Solution Approach 1:
The patent merges multiple small input blocks into compressed buffers before compression. By combining adjacent input blocks that can be compressed together, the system creates larger effective compression units that achieve better compression ratios while still maintaining the benefits of small input block processing for deduplication purposes.
Solution Approach 2:
The patent performs preliminary grouping of adjacent input blocks based on their compressibility characteristics before applying compression. This preliminary action identifies which small blocks should be combined together, allowing the compression algorithm to work on optimally-sized data units that balance compression efficiency with the storage benefits of compression.
Data Source
AI summary
Systems, methods, and computer readable storage mediums for optimistically managing compressed data in a storage system. When possible, multiple input blocks are compressed into a buffer and stored in a single physical block on a storage device. The metadata of the multiple input blocks can be managed separately. A fingerprint of the compressed buffer can be generated and used as an index to the single physical block. Alternatively, fingerprints of the uncompressed input blocks can be generated, and reference counts can be maintained to track the number of input blocks which reference the compressed buffer.