Resilient Distributed Storage via Erasure Coding and Virtual Zones
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Distributed replicated data storage systems face challenges in providing resilience and cost-effectiveness, as full replication methods are costly and require excessive storage capacity, while existing solutions do not efficiently manage data redundancy and accessibility across geographically dispersed storage zones.
Innovation Solution
The implementation of a resilient distributed replicated data storage system that partitions data into smaller objects and parity objects, using erasure coding techniques like Reed-Solomon, to reduce storage capacity requirements and ensure data accessibility by recreating data from available objects and parity objects, even if one zone becomes impaired, without the need for full replication across all zones.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If full replication is used to ensure data accessibility and resilience, then data availability is improved, but storage capacity requirements and costs increase significantly
Solution Approach 1:
The system segments data into smaller data objects that can be independently stored and managed across different storage zones. This segmentation allows for more flexible replication strategies where not every object needs to be fully replicated to every zone, thereby reducing total storage capacity requirements while maintaining data accessibility.
Solution Approach 2:
The system changes the replication parameter from full replication (100% copies to all zones) to partial replication based on data object priority and access patterns. By adjusting replication factors and selecting which data objects to replicate to which zones, the system maintains data accessibility for critical objects while reducing overall storage capacity requirements.
2Quantity of substance
If data is partitioned and parity objects are used to reduce storage capacity, then storage costs are reduced, but system complexity increases
Solution Approach 1:
The system creates parity objects as simplified copies that contain redundant information derived from the original data objects using erasure coding. These parity objects can be used to reconstruct lost data without requiring full copies of all data objects, thereby reducing storage capacity while managing complexity through standardized copying and reconstruction algorithms.
3Reliability
If erasure coding is implemented to recreate data from available objects, then data resilience is improved, but computational overhead and processing time increase
Solution Approach 1:
The system performs preliminary actions by pre-calculating and storing parity objects alongside data objects during the data writing phase. This preliminary computation of erasure coding parameters and parity information eliminates the need for complex real-time calculations during data reconstruction, thereby reducing processing time when data needs to be recreated from available objects.
Data Source
AI summary
A resilient distributed replicated data storage system is described herein. The storage system includes zones that are independent, and autonomous from each other. The zones include nodes that are independent and autonomous. The nodes include storage devices. When a data item is stored, it is partitioned into a plurality of data objects and a plurality of parity objects are calculated. Reassembly instructions are created for the data item. The data objects, parity objects and reassembly instructions are spread across nodes and zones in the storage system according to a policy for the data item. When a zone is inaccessible, a virtual zone is created and used until the intended zone is available. When a read request is received, the data item is prepared from the lowest latency nodes according to the reassembly instructions, and a virtual zone is accessed in place of a real zone when the real zone is inaccessible.