Meta Chunk Erasure Coding for Lower Storage Capacity Overhead
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Traditional storage area network (SAN) and network-attached storage (NAS) architectures face challenges in managing and protecting large multi-petabyte data capacities, leading to unreasonably high capacity overheads due to the immutable nature of sealed chunks with reduced data fragments in object storage systems like Elastic Cloud Storage (ECS™).
Innovation Solution
The system generates meta chunks by combining source chunks with fewer data fragments, employing erasure coding to create coding fragments that can recover the source chunks during failure conditions, thereby reducing capacity overheads and enhancing data protection efficiency.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If source chunks with fewer data fragments are individually protected with erasure coding, then data protection reliability is improved, but capacity overhead increases unreasonably high
Solution Approach 1:
The patent combines multiple source chunks with fewer data fragments into a single meta chunk before applying erasure coding. This merging approach allows the system to protect multiple chunks simultaneously with a single set of coding fragments, dramatically reducing capacity overhead while maintaining data protection reliability across all combined chunks.
2Reliability
If traditional verification procedures are implemented for data protection, then data integrity is improved, but resource consumption increases significantly
Solution Approach 1:
The patent implements a self-service verification mechanism where the system automatically verifies data integrity through the meta chunk structure and coding fragments without requiring external verification procedures. This eliminates resource-demanding manual or external verification steps while maintaining data integrity through automated checks embedded in the meta chunk protection mechanism.
3Reliability
If data protection is enhanced for chunks with reduced data fragments, then data security is improved, but data access efficiency deteriorates
Solution Approach 1:
The patent segments the protection mechanism by introducing meta chunks that group multiple source chunks together. This segmentation allows the system to apply protection at a higher level (meta chunk level) rather than individually to each source chunk, reducing the overhead and improving data access efficiency while maintaining security through the combined protection structure.
Data Source
AI summary
Data protection with meta chunks increases capacity use efficiency without verification and data copying. In one aspect, a meta chunk is a data protection unit, which combines two or more source chunks that are determined to have a reduced sets of data fragments. The meta chunk can be encoded to generate a set of coding fragments, which can be stored and utilized to recover data fragments of any of the two or more source chunks. Further, the source chunks can be linked to the meta chunk. Furthermore, the sets of coding fragments, that were previously generated by individually encoding each source chunk, can be deleted.