Erasure Coding Across Regions for Cloud Object Storage Durability
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Cloud object storage platforms face a significant risk of object loss as the number of objects increases, despite high durability guarantees, due to correlated failures within regions, and existing solutions like multi-region data mirroring incur high storage costs.
Innovation Solution
Implementing erasure coding across multiple regions by grouping data objects into virtual segments, computing parity information, and uploading these segments in a striped fashion across different regions, along with a novel garbage collection process using a master-slave configuration to minimize network bandwidth costs.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If multi-region data mirroring is used to reduce object loss risk, then object durability is improved, but storage costs increase significantly
Solution Approach 1:
The system segments data into virtual segments and applies erasure coding to divide data into fragments and parity information, distributing them across multiple regions. This segmentation approach reduces storage requirements compared to full mirroring while maintaining durability through distributed redundancy.
Solution Approach 2:
The system changes the redundancy parameter from full mirroring (100% duplicate storage) to erasure coding with configurable parity ratios. By adjusting the number of data fragments and parity objects, the system optimizes the balance between storage efficiency and object durability, achieving high reliability at lower storage costs.
2Quantity of substance
If the number of stored objects increases to meet customer backup needs, then data capacity is improved, but the likelihood of at least one object loss increases
Solution Approach 1:
The system merges multiple objects into virtual segments for collective erasure coding protection. Instead of protecting each object independently, grouping objects into segments allows shared redundancy resources, improving overall system durability while scaling to handle large numbers of objects efficiently.
Solution Approach 2:
The system transitions from protecting individual objects in isolation to protecting objects through their segment relationships across multiple regions. This dimensional shift from object-level to segment-level protection enables scalable durability that maintains reliability even as the total number of objects increases.
3Reliability
If erasure coding is applied within a single region, then object durability is improved, but the system remains vulnerable to correlated failures within that region
Solution Approach 1:
The system segments virtual segments across multiple geographic regions, ensuring that erasure coding fragments are distributed beyond a single region. This segmentation strategy isolates failures to specific regions, preventing correlated failures from affecting the entire dataset and enabling reconstruction from healthy regions.
Solution Approach 2:
The system introduces region distribution as an intermediary layer between erasure coding and physical storage. By placing parity objects and data fragments across multiple regions, the system mediates against regional failures, allowing data recovery even when entire regions experience correlated failures.
Data Source
AI summary
Techniques for using erasure coding across multiple regions to reduce the likelihood of losing objects in a cloud object storage platform are provided. In one set of embodiments, a computer system can upload each of a plurality of data objects to each of a plurality of regions of the cloud object storage platform. The computer system can further compute a parity object based on the plurality of data objects, where the parity object encodes parity information for the plurality of data objects. The computer system can then upload the parity object to another region of the cloud object storage platform different from the plurality of regions.