Distributed Data Storage with Two-Level Error Correction
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing distributed storage systems face high network traffic and storage overhead due to the need for redundant data replication, which can lead to network congestion and increased costs, especially in large-scale applications like social networking and search engines, where data must be accessible despite disruptions.
Innovation Solution
A distributed data storage system that generates redundant level one error correction blocks (L1EC) and level two error correction blocks (L2EC) to enable local data reconstruction without relying on network communication, allowing for efficient fault-tolerant storage across multiple locations.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If full redundant copies of data are maintained in multiple locations, then data reliability and accessibility are improved, but storage overhead becomes prohibitive
Solution Approach 1:
The patent extracts only the essential redundancy information needed for data recovery rather than storing complete data copies. By computing and storing only recovery codes (parity information) instead of full data replicas, the system achieves fault tolerance with significantly reduced storage overhead, directly resolving the contradiction between reliability and storage quantity.
Solution Approach 2:
The patent creates simplified copies in the form of recovery codes rather than complete data copies. These recovery codes are mathematical transformations of the original data that enable reconstruction of lost data without requiring full replicas, thus achieving the copying function with minimal storage overhead.
2Reliability
If recovery codes are distributed across separate data storage locations, then fault tolerance against data disruption events is improved, but network traffic during data reconstruction increases
Solution Approach 1:
The patent segments the recovery system into two distinct components: distributed recovery codes stored at remote locations and local data pieces stored at the primary location. This segmentation allows the system to tolerate remote storage failures while minimizing network traffic by enabling local reconstruction when only local data is lost, and efficient remote reconstruction when remote data is needed.
Solution Approach 2:
The patent introduces an intermediary mechanism where recovery codes serve as mediators between distributed data pieces. These recovery codes enable direct local reconstruction without requiring communication with remote storage locations, thus reducing network traffic while maintaining fault tolerance capabilities.
3Speed
If all recovery codes and data pieces are stored locally, then data reconstruction speed is improved, but fault-tolerance characteristics are adversely affected
Solution Approach 1:
The patent applies local quality by storing complete data and recovery codes locally at the primary storage location, enabling fast local reconstruction. Simultaneously, distributed copies of recovery codes are placed at remote locations to provide fault tolerance against local failures. This creates different quality characteristics at different locations: full capability locally for speed, and distributed redundancy remotely for reliability.
Data Source
AI summary
Fault-tolerant storage is provided using a distributed data storage system that receives input data from clients and divides that data into data blocks for storage. The data blocks are processed using a coding scheme that generates redundant level one error correction blocks (L1EC Blocks). The L1EC blocks enable the reconstruction of one or more damaged or inaccessible data blocks, and the L1EC blocks and the data blocks are divided into distribution sets and stored at a plurality of data storage locations. At each data storage location additional level two error correction blocks (L2EC blocks) are generated that provide local data redundancy. Upon detecting a data disruption event, an inaccessible data storage location is identified and the elements that were stored at the inaccessible data storage location are reconstructed.