Mid-Size Object Erasure Coding Without Preliminary Mirroring
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Contemporary cloud-based data storage systems face inefficiencies in handling mid-size objects due to the need for preliminary protection schemes, which result in increased data traffic and metadata requirements, complicating the prediction of when chunks will become sealed and leading to delayed data acknowledgement.
Innovation Solution
Implementing direct erasure coding for mid-size objects without a preliminary protection scheme, where objects are divided into smaller fragments and coding fragments, which are then written directly into chunk data and coding fragments, reducing data traffic and metadata needs.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If triple mirroring is used as a preliminary protection scheme before erasure coding, then data reliability is improved, but internal data traffic increases significantly
Solution Approach 1:
The patent applies preliminary action by performing erasure coding immediately when data arrives at the storage system, before writing to chunks. This eliminates the need for triple mirroring as a preliminary protection scheme, as the data is already protected via erasure coding at the time of write. The system divides incoming data into segments, applies erasure coding to generate protected segments, and writes them directly to chunks, thereby avoiding the additional data traffic generated by creating and managing mirror copies.
2Reliability
If triple mirroring is used as a preliminary protection scheme, then data reliability is improved, but the complexity of the protection scheme increases
Solution Approach 1:
The patent extracts the protection function from the chunk sealing process and applies it immediately to incoming data. By separating the protection mechanism (erasure coding) from the chunk management process, the system eliminates the need for complex coordination between mirroring and erasure coding operations. The protection is applied as a standalone step before data enters the chunk workflow, simplifying the overall protection scheme while maintaining reliability.
3Reliability
If delayed erasure coding is implemented with triple mirroring, then data protection is ensured, but data acknowledgement delay occurs
Solution Approach 1:
The patent applies preliminary action by performing erasure coding immediately when data arrives at the storage system, before writing to chunks. This eliminates the need for triple mirroring as a preliminary protection scheme, as the data is already protected via erasure coding at the time of write. The system divides incoming data into segments, applies erasure coding to generate protected segments, and writes them directly to chunks, thereby avoiding the additional data traffic generated by creating and managing mirror copies.
4Loss of energy
If dedicated Type-II chunks are allocated for large objects, then data traffic is reduced, but applicability to mid-size objects is limited
Solution Approach 1:
The patent applies parameter changes by introducing a size threshold parameter that determines whether erasure coding is applied immediately or via triple mirroring. Objects at or above this threshold (including mid-size objects) receive immediate erasure coding protection, while smaller objects use the traditional triple mirroring approach. This parameter-based differentiation allows the system to optimize data traffic for mid-size objects without sacrificing the proven reliability of triple mirroring for smaller objects, thereby expanding the applicability of the optimized path.
Data Source
AI summary
The disclosed technology generally describes a data protection scheme that for “mid-size” objects directly writes divided object data fragments, and performs erasure coding to directly write object coding fragments, to distributed storage locations in a node cluster. A storage container such as a chunk allocated for mid-size objects is distributed among the storage cluster nodes. When a mid-size object (e.g., between 24 megabytes and 128 megabytes) is to be created, the object data is divided into object data fragments and encoded into object coding fragments, with the data object fragments and object coding fragments written/appended to the distributed storage locations, without needing a preliminary protection scheme.