Erasure Coding Write Groups With Size-Matched Compressed Blocks
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing data protection methods, such as erasure coding, face inefficiencies due to the need for parity blocks that are as large as the largest data block in a group, which negates the benefit of compression, especially when dealing with varying sizes of compressed data blocks from unpredictable client workload patterns.
Innovation Solution
A technique that maintains a pool of varying-sized compressed data blocks and selects blocks for erasure coding based on size similarity, minimizing padding and optimizing parity block size by forming write groups with blocks close in size, using a best-match algorithm and biasing towards older blocks.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If erasure coding is applied to data blocks of varying sizes, then data protection is provided, but the parity block size must match the largest data block requiring excessive padding that negates compression benefits
Solution Approach 1:
The system performs preliminary compression on data blocks before erasure coding, and maintains a pool of pre-compressed blocks of various sizes. When forming write groups, the system pre-selects blocks with similar compression ratios to minimize padding requirements, thereby preserving compression benefits while enabling erasure coding for data protection.
Solution Approach 2:
The system dynamically adjusts write group composition based on compression ratio parameters. By selecting data blocks with similar compression ratios within acceptable thresholds, the system optimizes the balance between parity block size and compression efficiency, preventing excessive padding while maintaining data protection through erasure coding.
2Loss of substance
If data blocks are compressed to save storage capacity, then storage efficiency improves, but compressed blocks have varying sizes making erasure coding inefficient
Solution Approach 1:
The system changes the parameter selection criteria for write groups from strict size uniformity to similarity-based selection. By accepting blocks with comparable rather than identical sizes, the system maintains compression efficiency while enabling practical erasure coding without excessive padding overhead.
Solution Approach 2:
The system applies different quality requirements to different blocks within a write group. Rather than requiring all blocks to meet a strict size uniformity standard, the system allows local variations in block size while maintaining overall group coherence through similarity thresholds, preserving both compression benefits and erasure coding functionality.
3Ease of manufacture
If smaller data blocks are zero-padded to match the largest block size, then erasure coding can be applied uniformly, but the parity block size increases negating compression benefits
Solution Approach 1:
The system changes the approach from fixed block size padding to dynamic similarity-based grouping. By adjusting the selection parameters to favor blocks with comparable sizes, the system reduces padding requirements while maintaining uniform erasure coding application across the write group.
Solution Approach 2:
The system performs preliminary filtering and sorting of compressed blocks by size before forming write groups. This preliminary action ensures that blocks selected for the same write group have similar sizes, minimizing the padding required for uniform erasure coding while preserving compression efficiency.
Data Source
AI summary
A technique provides efficient data protection, such as erasure coding, for data blocks of volumes served by storage nodes of a cluster. Data blocks associated with write requests of unpredictable client workload patterns may be compressed. A set of the compressed data blocks may be selected to form a write group and an erasure code may be applied to the group to algorithmically generate one or more encoded blocks in addition to the data blocks. Due to the unpredictability of the data workload patterns, the compressed data blocks may have varying sizes. A pool of the various-sized compressed data blocks may be established and maintained from which the data blocks of the write group are selected. Establishment and maintenance of the pool enables selection of compressed data blocks that are substantially close to the same size and, thus, that require minimal padding.