Compressed Data Layout for Deduplication-Safe Container Compression
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Current storage environments face inefficiencies in data compression and deduplication, as existing methods often require metadata management and impact deduplication savings, and do not allow for flexible compression of different data types or variable chunk sizes without affecting deduplication efficiency.
Innovation Solution
A layout format for compressed data that enables container-level compression using variable chunk sizes and different algorithms, storing metadata within disk blocks to maintain self-sufficiency of each data block for decompression, allowing for flexible grouping and compression of similar data types without impacting deduplication savings.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Loss of substance
If container-level compression is implemented, then storage efficiency is improved, but metadata management complexity increases
Solution Approach 1:
The patent extracts metadata from external management structures and embeds it directly within each compressed data block. This allows each block to be self-sufficient for decompression without requiring external metadata tracking, thereby reducing metadata management complexity while maintaining compression efficiency.
Solution Approach 2:
The patent segments the compressed data into fixed-size chunks with metadata embedded in each segment. This segmentation allows independent processing of each block without managing large-scale metadata structures, reducing overall system complexity while achieving container-level compression benefits.
2Adaptability or versatility
If variable chunk sizes are used for compression, then compression flexibility is improved, but deduplication efficiency deteriorates
Solution Approach 1:
The patent changes the chunk size parameter from variable to fixed, which simplifies deduplication operations while maintaining adequate compression flexibility through container-level processing. This parameter change resolves the contradiction by prioritizing deduplication efficiency without sacrificing all compression adaptability.
Solution Approach 2:
The patent applies different processing qualities at different levels: fixed-size chunks for efficient deduplication at the block level, while allowing variable compression algorithms and parameters at the container level. This multi-level quality approach maintains both deduplication efficiency and compression flexibility.
3Loss of information
If compression metadata is stored externally, then decompression information is preserved, but storage overhead increases
Solution Approach 1:
The patent merges metadata with the compressed data block by embedding it directly within the block structure. This combination eliminates the need for separate external metadata storage, reducing storage overhead while ensuring decompression information is always available with the data it describes.
4Ease of manufacture
If file-level compression is used, then implementation simplicity is maintained, but storage savings are limited
Solution Approach 1:
The patent transitions from file-level to container-level compression, adding a new dimension of compression granularity. This allows compression to operate on multiple data blocks simultaneously while maintaining implementation simplicity through a unified compression interface, thereby achieving greater storage savings without significantly increasing complexity.
Data Source
AI summary
Techniques are provided for a layout format for compressed data. A first set of data blocks are grouped into a first group based upon a first frequency of access to the first set of data blocks. A second set of data blocks are grouped into a second group based upon a second frequency of access to the second set of data blocks. The first set of data blocks are compressed into a first compression group using a first compression algorithm. The second set of data blocks are compressed into a second compression group using a second compression algorithm.