Locally Decodable Data Slices for Distributed Storage Recovery
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing data storage and processing systems face challenges in efficiently managing and processing large volumes of data across distributed networks while ensuring data integrity and security, particularly in the presence of storage and network failures.
Innovation Solution
A distributed computing system that employs dispersed storage and task processing units, utilizing error encoding and decoding techniques to store and process data across geographically diverse locations, ensuring data integrity and security, and enabling efficient recovery from failures without the need for redundant copies.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If data is stored using traditional redundant copying methods, then data reliability is improved, but storage efficiency deteriorates due to duplicate data occupying additional space
Solution Approach 1:
The patent segments data into multiple data slices and distributes them across different storage locations. Each slice contains a portion of the original data, and the segmentation is combined with erasure coding to create redundant yet compact representations that improve reliability without requiring full duplicate copies of the entire dataset
Solution Approach 2:
The patent transforms data from its original form into encoded representations using erasure coding algorithms. This parameter change converts k data slices into n encoded slices (where n > k), allowing the system to tolerate up to n-k slice failures while maintaining storage efficiency by avoiding traditional redundant copying
2Reliability
If data is encoded using complex error correction codes, then data integrity is improved, but processing complexity deteriorates
Solution Approach 1:
The patent performs error correction encoding in advance during the data writing process. By pre-encoding data into erasure-coded slices before storage, the system eliminates the need for complex real-time decoding operations during read tasks, thereby reducing processing complexity while maintaining data integrity
Solution Approach 2:
The patent creates encoded copies of data slices that contain redundancy information. These encoded copies are stored distributedly and can be used to reconstruct original data even when some slices are lost or corrupted, achieving data integrity through pre-computed redundant representations rather than complex processing
3Reliability
If data slices are distributed across geographically diverse locations, then system availability is improved, but network overhead deteriorates
Solution Approach 1:
The patent implements a threshold-based read mechanism where only k out of n encoded slices are needed to reconstruct the original data. This partial action approach allows the system to tolerate slice losses and distribute data geographically while minimizing network overhead by not requiring retrieval of all distributed slices for every operation
Data Source
AI summary
A storage network operates by: encoding at least one data object in the storage network as encoded data slices that include one or more locally decodable code segments; detecting at least one storage error associated with storage of at least one of the encoded data slices in the storage network; and recovering the at least one of the encoded data slices utilizing the one or more locally decodable code segments.