Erasure-Coded Meta Chunks With Complementary Fragment Combining
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Traditional storage area network (SAN) and network-attached storage (NAS) architectures face challenges in supporting large multi-petabyte data storage and protection efficiently, leading to high capacity overheads due to the immutable nature of sealed chunks in object storage systems like ECS™, which requires resource-demanding complete data re-protection for meta chunk generation.
Innovation Solution
The system modifies data fragments and their indices to generate complementary chunks that can be combined into meta chunks without performing a full erasure coding operation, reducing capacity overheads by summing coding fragments and allowing efficient data protection and recovery.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If complete data re-protection is performed for meta chunk generation, then data protection reliability is improved, but capacity overhead and resource consumption increase
Solution Approach 1:
The patent applies partial action by performing erasure coding only on the new data chunk rather than on the entire meta chunk. The system combines existing protected chunks with newly encoded chunks, using only the necessary portion of encoding operations to achieve data protection, thereby reducing capacity overhead while maintaining reliability
Solution Approach 2:
The system performs preliminary actions by pre-computing and storing coding fragments for existing chunks before meta chunk creation. These pre-computed coding fragments are then reused during meta chunk generation, eliminating the need for complete re-protection and reducing both capacity overhead and computational resources required
2Reliability
If complete data re-protection is performed for meta chunk generation, then data protection reliability is improved, but processing time and resource demand increase
Solution Approach 1:
The patent performs erasure coding only on the new data chunk rather than the entire meta chunk, significantly reducing processing time. The system leverages existing protected data and applies encoding operations only where necessary, maintaining data protection reliability while minimizing time loss
Solution Approach 2:
The system performs preliminary encoding operations and stores coding fragments in advance. During meta chunk generation, these pre-computed fragments are reused, eliminating the need for time-consuming re-encoding of entire datasets and reducing processing time while maintaining protection reliability
3Stability of the object's composition
If chunks are sealed with immutable content, then data integrity is improved, but flexibility for capacity optimization is reduced
Solution Approach 1:
The patent merges multiple sealed immutable chunks into a meta chunk structure. By combining these immutable units with newly encoded data, the system maintains the integrity benefits of sealed chunks while achieving capacity optimization through flexible meta chunk creation and management
Solution Approach 2:
The system performs preliminary organization and validation of sealed chunks before incorporating them into meta chunks. This preliminary action ensures data integrity is maintained while enabling flexible capacity management through the meta chunk abstraction layer
Data Source
AI summary
Resource-efficient data protection is performed by generating meta chunks in storage systems that utilize erasure coding. During erasure coding with a k+m configuration, a data chunk can be divided into k data fragments, having indices 1 to k, that can be encoded by combining them with corresponding coefficients of a coding matrix, to generate coding fragments. Source portions that have a reduced set (e.g., less than k data fragments) of data fragments can be modified such that they are made complementary (e.g., that do not have common indices) without complete data re-protection. The complementary portions can then be combined to generate a meta chunk. The coding fragments of the complementary portions can be added to generate coding fragments for the meta chunk, which can then be utilized to recover data fragments of any of the source portions.