Erasure Coding Write Groups With Pooled Compressed Blocks
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Erasure coding in storage clusters faces inefficiencies due to varying sizes of compressed data blocks, which require padding to match the largest block size, negating the benefits of compression and increasing storage needs.
Innovation Solution
A pool of varying-sized compressed data blocks is maintained, allowing for the selection of blocks that are close in size to form write groups, minimizing the need for padding and enabling efficient generation of parity blocks through a best-match algorithm.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Quantity of substance
If data blocks are compressed to increase storage capacity, then storage efficiency is improved, but the varying sizes of compressed blocks require padding to match the largest block size, which negates the compression benefit and increases storage needs
Solution Approach 1:
The patent pools compressed data blocks of various sizes before forming write groups for erasure coding. By pre-collecting and organizing blocks in a pool, the system can later select blocks with similar sizes, minimizing the padding required when forming write groups and maintaining compression efficiency.
Solution Approach 2:
The patent dynamically adjusts the composition of write groups by selecting data blocks with similar sizes from the pooled blocks. This parameter optimization (matching block sizes) reduces the padding overhead while maintaining the benefits of compression and erasure coding for data protection.
2Adaptability or versatility
If blocks are padded to match the largest block size for erasure coding, then parity calculation compatibility is improved, but the parity block size increases, reducing storage efficiency
Solution Approach 1:
The system performs preliminary pooling and sizing-matching of data blocks before the parity calculation phase. This advance preparation ensures that when erasure coding occurs, the blocks are already optimized for minimal padding, making the parity calculation process both compatible and efficient.
Solution Approach 2:
The patent applies local optimization by selecting and grouping blocks with locally similar sizes within the pool. This localized size-matching approach ensures that each write group contains blocks of comparable size, minimizing padding requirements while maintaining the adaptability needed for erasure coding operations.
3Adaptability or versatility
If compressed data blocks of varying sizes are used directly in write groups, then storage flexibility is improved, but padding is required to match block sizes, increasing the number of parity blocks needed
Solution Approach 1:
The patent establishes a pool of compressed data blocks in advance, organizing them by size characteristics. This preliminary organization maintains the flexibility to handle varying data sizes while enabling efficient selection of similarly-sized blocks for write groups, thereby reducing padding requirements and improving overall storage efficiency.
Solution Approach 2:
The system dynamically selects blocks from the pool based on size matching criteria when forming write groups. This dynamic selection process maintains adaptability to varying data workloads while optimizing for minimal padding, thus preserving both flexibility and productivity in storage operations.
Data Source
AI summary
A technique provides efficient data protection, such as erasure coding, for data blocks of volumes served by storage nodes of a cluster. Data blocks associated with write requests of unpredictable client workload patterns may be compressed. A set of the compressed data blocks may be selected to form a write group and an erasure code may be applied to the group to algorithmically generate one or more encoded blocks in addition to the data blocks. Due to the unpredictability of the data workload patterns, the compressed data blocks may have varying sizes. A pool of the various-sized compressed data blocks may be established and maintained from which the data blocks of the write group are selected. Establishment and maintenance of the pool enables selection of compressed data blocks that are substantially close to the same size and, thus, that require minimal padding.