Key-Blob Archive Data Recovery via Erasure Coding
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing data storage systems face challenges in providing reliable, scalable, and fault-tolerant storage for large binary objects across geographically distributed data centers, especially in scenarios involving network partitions and data center outages, while also requiring efficient retrieval and redundancy management.
Innovation Solution
A scalable key-blob archive (KBA) system that utilizes erasure correction coding, replication, and self-healing features to ensure reliable storage and retrieval of large binary objects, with a messaging protocol for Put and Get operations, convergence protocols for metadata and fragment recovery, and scrubbing protocols for maintaining data integrity.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If data is replicated across geographically distributed data centers, then data availability and fault tolerance are improved, but system complexity and data synchronization difficulty increase
Solution Approach 1:
The patent segments data into fragments and distributes them across multiple data centers. Each data center stores a portion of the fragmented data, allowing the system to achieve fault tolerance and availability without requiring complete data replication at every location. This segmentation approach reduces the complexity of full data synchronization while maintaining reliability.
Solution Approach 2:
The patent introduces erasure coding as an additional dimension of data protection beyond simple replication. By applying error correction codes across fragmented data distributed geographically, the system achieves fault tolerance through a mathematical framework that operates independently of physical data center locations, thereby managing complexity through abstraction.
2Reliability
If erasure correction coding is used for data redundancy, then data recovery capability is improved, but processing time and computational resources increase
Solution Approach 1:
The patent applies erasure correction coding during the initial data storage phase rather than during recovery operations. By pre-processing data with error correction codes before distribution, the system prepares recovery capability in advance, so that when data loss occurs, the recovery process simply involves reassembling fragments using pre-computed parity information, significantly reducing recovery time.
3Adaptability or versatility
If data fragments are distributed across multiple storage nodes, then system scalability is improved, but data retrieval complexity and network overhead increase
Solution Approach 1:
The patent implements a coordination mechanism where storage nodes provide feedback about data fragment location and availability. This feedback system enables the central coordinator to manage data retrieval operations efficiently by knowing exactly where fragments are stored, reducing the complexity of distributed data access while maintaining scalability.
Data Source
AI summary
A method of recovering a value in a data storage system includes maintaining a list of entries corresponding to values that have been inserted into the data storage system, determining whether the values are at an At Maximum Redundancy (AMR) state, and if not, the converging the values to the AMR state.