Erasure-Coded Meta Chunks for Lower Storage Capacity Overhead
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Traditional storage area network (SAN) and network-attached storage (NAS) architectures face challenges in supporting large multi-petabyte data storage and protection, leading to unreasonably high capacity overheads due to the immutability of sealed chunks in object storage systems like Elastic Cloud Storage (ECS™), which requires resource-demanding complete data re-protection for erasure-coded meta chunks.
Innovation Solution
The system selects source chunks with fewer data fragments, divides them into indexed data fragments, and applies erasure coding to generate source coding fragments, then combines these to create meta chunks with combined coding fragments that can recover data during failure conditions, reducing capacity overheads without complete data re-protection.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If complete data re-protection is performed for erasure-coded meta chunks, then data reliability is improved, but capacity overhead increases unreasonably and resource consumption increases
Solution Approach 1:
The patent combines multiple source chunks into a single meta chunk, merging their coding fragments. Instead of maintaining separate protection sets for each source chunk, the system creates a unified protection set at the meta chunk level, reducing redundant storage overhead while maintaining data reliability through erasure coding.
Solution Approach 2:
The meta chunk serves multiple functions: it consolidates data from multiple source chunks, provides unified erasure-coded protection for all source chunks, and enables centralized management of coding fragments. This multi-functionality reduces the overall capacity overhead compared to individual protection sets for each source chunk.
2Reliability
If complete data re-protection is performed for erasure-coded meta chunks, then data protection is improved, but resource consumption increases
Solution Approach 1:
By merging the protection mechanisms of multiple source chunks into a single meta chunk protection set, the system reduces the total computational resources required for encoding and managing protection. Instead of independently processing protection for each source chunk, the unified approach processes protection once at the meta chunk level.
3Stability of the object's composition
If sealed chunks are made immutable, then data integrity is improved, but flexibility in data management deteriorates
Solution Approach 1:
The system performs preliminary actions by creating erasure-coded protection sets and meta chunks before source chunks are sealed and made immutable. This allows data consolidation and protection setup to occur while data is still accessible, maintaining integrity after sealing without sacrificing future management flexibility through the meta chunk abstraction layer.
Data Source
AI summary
Resource-efficient data protection is performed by generating meta chunks in storage systems that utilize erasure coding. During erasure coding with a k+m configuration, a data chunk can be divided into k data fragments, having indices 1 to k, that can be encoded by combining them with corresponding coefficients of a coding matrix, to generate coding fragments. Source portions that have a reduced set (e.g., less than k data fragments) of data fragments and that are complementary (e.g., that do not have common indices) can be determined and combined to generate a meta chunk. The coding fragments of the source portions can be added to generate coding fragments for the meta chunk, which can then be utilized to recover data fragments of any of the source portions. Further, the coding fragments, that were previously generated by individually encoding each source portion, can be deleted.