Distributed Data Storage Using Erasure Coding Across Failure Zones
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Distributed replicated data storage systems face high costs due to the need for full replication of data across multiple zones, which is costly and inefficient, especially when all data must be replicated at all zones, leading to increased storage media requirements.
Innovation Solution
A resilient distributed replicated data storage system that partitions data into smaller objects and parity objects, using erasure coding techniques like Reed-Solomon, allowing for data recreation from any combination of available objects, reducing the need for full replication and minimizing storage capacity, thereby lowering costs while maintaining data accessibility and reliability.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If full replication of data across multiple zones is implemented, then data resiliency and accessibility are improved, but storage costs and storage media requirements increase significantly
Solution Approach 1:
The patent segments data into smaller objects and further divides these objects into multiple parts using erasure coding techniques. Instead of replicating entire data objects across all zones, the system breaks down data objects into smaller segments and distributes these segments across different zones. This segmentation allows the system to achieve the same resiliency with fewer storage resources, as any sufficient combination of segments from different zones can reconstruct the original data.
Solution Approach 2:
The patent changes the fundamental parameter of data storage from full replication to erasure-coded segmentation. By transforming data into encoded segments with redundant information distributed across zones, the system alters how resiliency is achieved. This parameter change enables the system to maintain data accessibility and durability while reducing the total storage capacity required, as the encoded segments contain distributed redundancy rather than requiring complete duplicate copies.
2Quantity of substance
If data is partitioned into smaller objects and parity objects using erasure coding, then storage capacity requirements are reduced, but system complexity increases
Solution Approach 1:
The patent implements self-service mechanisms where the distributed storage system automatically performs erasure encoding, segment distribution, and data reconstruction operations without requiring manual intervention. The system autonomously manages the complexity of partitioning data into objects and further into encoded segments, handling the mathematical transformations and redundancy calculations automatically. This self-service approach masks the underlying system complexity from users while achieving reduced storage capacity requirements through sophisticated erasure coding techniques.
Data Source
AI summary
A failure resilient distributed replicated data storage system is described herein. The storage system includes zones that are independent, and autonomous from each other. The zones include nodes that are independent and autonomous. The nodes include storage devices. When a data item is stored, it is partitioned into a plurality of data objects and a plurality of parity objects calculated. Reassembly instructions are created for the data item. The data objects and parity objects are spread across all nodes and zones in the storage system. Reassembly instructions are also spread across the zones. When a read request is received, the data item is prepared from the lowest latency nodes according to the reassembly instructions. This provides for data resiliency while keeping the amount of storage space required relatively low.