Opportunistic Compression for Storage Efficiency
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
The transition to 4 KiB object sizes in computing systems diminishes the effectiveness of data compression, leading to reduced storage efficiency and network bandwidth optimization, as devices with 4 KiB native sector sizes incur overhead for smaller sector sizes and limit compression effectiveness on small data objects.
Innovation Solution
The technique involves opportunistically compressing data by merging adjacent 4 KiB objects into 64 KiB objects when possible, particularly during large writes, sequential workloads, and remote backups, using a storage system with a Merkle tree structure and look-aside bits to manage compression and decompression efficiently.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Quantity of substance
If data is compressed into smaller sizes to improve storage efficiency, then storage efficiency improves, but device complexity increases due to overhead for emulating smaller sector sizes
Solution Approach 1:
The patent segments data into fixed 4 KiB objects that align with native sector sizes, avoiding the need to compress individual smaller objects. This segmentation strategy allows compression to operate at the object level rather than at the sector level, eliminating the overhead of emulating smaller sectors while maintaining storage efficiency through compression of complete 4 KiB objects.
Solution Approach 2:
The patent changes the compression parameter from operating on variable-sized sectors to operating on fixed 4 KiB objects. By standardizing the object size to match the native sector size, the system achieves compression efficiency without the device complexity overhead associated with emulating smaller sector sizes.
2Quantity of substance
If compression is applied to 4 KiB objects to improve storage efficiency, then storage efficiency improves, but compression effectiveness diminishes
Solution Approach 1:
The patent merges multiple 4 KiB objects into larger compressed units, particularly during large writes and sequential workloads. By combining adjacent 4 KiB objects into 64 KiB or larger compressed blocks, the system maintains high compression effectiveness while working with the 4 KiB object constraint. This merging approach allows compression algorithms to process larger data volumes more effectively.
Solution Approach 2:
The patent dynamically adjusts compression behavior based on workload characteristics. During sequential workloads and large writes, the system merges objects to enable more effective compression. For random access patterns, it maintains individual 4 KiB object handling. This dynamic adaptation ensures compression effectiveness is optimized for the current operational context.
3Quantity of substance
If compression is applied to small data objects to improve storage efficiency, then storage efficiency improves, but compression/decompression overhead increases
Solution Approach 1:
The patent applies compression selectively rather than universally. It compresses data when and where it provides benefit, such as during large writes and sequential workloads, while avoiding compression for small or randomly accessed objects where the overhead would exceed the benefits. This partial application strategy reduces unnecessary compression/decompression overhead while maintaining storage efficiency where compression is effective.
Data Source
AI summary
A method for storing data in a storage system, includes opportunistically compressing a plurality of objects of a first size during a write operation, storing the compressed objects of the first size if they compress acceptably, and storing other objects of a second size uncompressed. It further includes determining during a read operation whether an object of the second size is stored as a part of a compressed object; if the object of the second size is not stored as a part of a compressed object of the first size, then reading the object of the second size from storage; if the object of the second size is stored as a part of the compressed object of the second size, then: reading the compressed object of the first size from storage; uncompressing the compressed object of the first size; and extracting the object of the second size.