Chunk-Container Erasure Coding for Small-Write Fault Tolerance
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing erasure coding techniques face inefficiencies in storage space utilization and resource overhead due to the need for post-process operations when handling small data writes and data stripe width adjustments, particularly in scenarios with increased fault tolerance requirements.
Innovation Solution
The technique employs container-based erasure coding by utilizing entire replicated chunk files as data stripes, forming logical grouping containers based on specific criteria, reducing the need for post-process operations and resource usage.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If traditional erasure coding is applied to small data writes, then fault tolerance is improved, but storage space utilization deteriorates due to inefficient data stripe width allocation
Solution Approach 1:
The patent segments data into smaller chunk files with individual erasure coding protection rather than applying EC to larger data stripes. Each chunk file is independently protected with its own parity chunks, allowing flexible allocation that matches actual data write sizes and improves storage efficiency while maintaining fault tolerance.
Solution Approach 2:
The system dynamically adjusts the number of parity chunks and data stripe configuration based on the size and characteristics of incoming data writes. This dynamic adaptation allows optimal storage space utilization for different data sizes while maintaining consistent fault tolerance protection.
2Productivity
If data stripe width is increased to reduce overhead, then write efficiency is improved, but storage space efficiency deteriorates for small data writes
Solution Approach 1:
The patent divides data into smaller chunk files that can be independently processed and protected. This segmentation allows the system to use smaller, more efficient data stripe widths for small data writes without the overhead penalty of large stripe configurations, thereby improving both write efficiency and storage space efficiency.
Solution Approach 2:
The system changes the erasure coding parameters (number of parity chunks, data stripe width) based on the size of the data being written. For small data writes, the system uses reduced parameters that minimize overhead while still providing adequate protection, thus improving storage space efficiency without sacrificing write efficiency.
3Reliability
If post-process erasure coding operations are performed, then data protection is improved, but resource overhead increases due to data chunk relocation and metadata updates
Solution Approach 1:
The patent applies erasure coding protection to data chunks as they are being written to chunk files, rather than performing post-process operations. This preliminary application of EC protection eliminates the need for subsequent data chunk relocation and metadata updates, thereby reducing resource overhead while maintaining data protection.
Solution Approach 2:
The system performs erasure coding operations self-contained within the chunk file writing process itself. Each chunk file is independently protected using its own parity chunks, eliminating the need for complex post-process operations that would require coordinating multiple data chunks and updating metadata across the system.
Data Source
AI summary
A repository of replicated chunk files is analyzed to identify chunk files that meet at least a portion of combination criteria. Selected chunk files are associated together under a data protection grouping container. Erasure coding is applied to the data protection grouping container including by utilizing the selected chunk files as different data stripes of the erasure coding and generating one or more parity stripes based on the different data stripes.