Erasure-Coded Multi-Zone Storage for Resilient Data Access
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Distributed replicated data storage systems face high costs and inefficiencies due to the need for full replication of data across multiple zones, which can be costly and resource-intensive, especially when all data must be replicated at all zones for resiliency.
Innovation Solution
A resilient distributed replicated data storage system that partitions data into smaller objects and parity objects, using erasure coding techniques like Reed-Solomon, allowing for data recreation from any available objects, reducing the need for full replication and minimizing storage capacity, thereby lowering costs while maintaining resiliency and accessibility.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If full replication of data is performed across multiple zones, then data resiliency and accessibility are improved, but storage costs and resource consumption increase significantly
Solution Approach 1:
The patent segments data into smaller objects and distributes them across multiple zones using erasure coding. Instead of replicating entire data sets across all zones, the system divides data into fragments (e.g., 6 data objects) and stores them in different zones, allowing reconstruction of the original data from any sufficient subset of fragments. This segmentation enables resiliency without requiring full replication of all data in all zones.
Solution Approach 2:
The patent changes the storage parameter from full replication (100% data in each zone) to erasure coding with configurable redundancy ratios. By adjusting the number of data objects and parity objects (e.g., 6+3 configuration), the system can tune the balance between storage efficiency and data resiliency, reducing storage capacity requirements while maintaining acceptable reliability levels.
2Ease of operation
If full replication of data is performed across multiple zones, then data accessibility is improved, but system complexity and resource intensity increase
Solution Approach 1:
The patent introduces a metadata layer that acts as an intermediary between the client and distributed data objects. This metadata contains information about the location and status of data fragments across zones, enabling the system to intelligently retrieve data from available zones without requiring full replication. The metadata layer simplifies the complexity of distributed data access by providing a unified view of data locations.
3Quantity of substance
If less data is stored to reduce costs, then storage capacity is reduced, but data resiliency may be compromised
Solution Approach 1:
The patent implements erasure coding with parity objects that serve as a form of beforehand cushioning. By pre-calculating and storing parity information (e.g., 3 parity objects for 6 data objects), the system creates a safety buffer that enables data reconstruction even when some data objects are lost or inaccessible. This cushioning mechanism ensures resiliency is maintained even with reduced storage capacity compared to full replication.
Data Source
AI summary
A failure resilient distributed replicated data storage system is described herein. The storage system includes zones that are independent, and autonomous from each other. The zones include nodes that are independent and autonomous. The nodes include storage devices. When a data item is stored, it is partitioned into a plurality of data objects and a plurality of parity objects calculated. Reassembly instructions are created for the data item. The data objects and parity objects are spread across all nodes and zones in the storage system. Reassembly instructions are also spread across the zones. When a read request is received, the data item is prepared from the lowest latency nodes according to the reassembly instructions. This provides for data resiliency while keeping the amount of storage space required relatively low.