Encoded Data Slice Recovery Using Local Redundancy
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing data storage and processing systems face challenges in efficiently managing and recovering missing data in a distributed storage network without the need for redundant copies, while ensuring data integrity and security, especially in the face of failures and hacking attempts.
Innovation Solution
A distributed computing system that employs locally decodable redundancy data encoding and decoding techniques to store and retrieve data across geographically dispersed execution units, utilizing error correction schemes to rebuild missing data slices and ensure data integrity, while supporting secure and reliable data storage and processing.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If redundant copies of data are stored to ensure data recovery, then data reliability is improved, but storage space consumption increases
Solution Approach 1:
The patent segments data into multiple data slices and distributes them across different storage locations. Instead of storing complete redundant copies, the system divides data into fragments that can be independently stored and later reassembled, reducing overall storage requirements while maintaining recovery capability.
Solution Approach 2:
The system transforms data from its original form into encoded representations using error correction codes. By changing the parameter representation of data (from raw copies to encoded slices), the system achieves more efficient storage utilization while preserving the ability to reconstruct original data from fewer slices.
2Reliability
If data is distributed across multiple geographically dispersed locations, then system reliability and security are improved, but data recovery complexity increases
Solution Approach 1:
The patent implements self-healing capabilities where the system automatically detects missing or corrupted data slices and initiates recovery processes without human intervention. The error correction codes enable the system to self-reconstruct lost data from remaining slices, reducing operational complexity despite geographic distribution.
Solution Approach 2:
The system incorporates integrity verification mechanisms that continuously monitor the status of distributed data slices. When failures are detected, feedback loops trigger automatic recovery operations, managing the complexity of distributed recovery through structured monitoring and response protocols.
3Reliability
If error correction codes are used to rebuild missing data, then data integrity is improved, but processing time increases
Solution Approach 1:
The patent applies error correction encoding to data before distribution, preparing the data in advance for potential recovery scenarios. This preliminary encoding ensures that when data loss occurs, the recovery process can proceed efficiently using pre-computed redundancy information, reducing actual recovery time.
Solution Approach 2:
The system uses partial redundancy where only sufficient error correction codes are applied to enable recovery of a specific number of lost slices. Rather than providing excessive redundancy that would slow processing, the system calculates the minimum necessary encoding to recover up to a threshold number of missing slices, optimizing the balance between integrity and processing speed.
Data Source
AI summary
A processing system of a storage network operates by: sending, to at least one storage unit of the storage network, at least one read request corresponding to at least a read threshold number of a set of encoded data slices to be retrieved, wherein the set of encoded data slices correspond to data, wherein the data is coded in accordance with dispersed error coding parameters that include a write threshold number and the read threshold number, wherein the write threshold number is a number of encoded data slices in the set of encoded data slices and wherein the read threshold number is a number of the set of encoded data slices that is required to decode the data; receiving, at the at least one processing circuit and from the at least one storage unit, a first subset of the set of encoded data slices, wherein at least one missing encoded data slice was not included in the first subset and wherein a number of encoded data slices in the first subset is less than the read threshold number; generating, via the at least one processing circuit, at least one rebuilt encoded data slice corresponding to the at least one missing encoded data slice utilizing locally decodable redundancy data, wherein the locally decodable redundancy data corresponds to a second subset of the set of encoded data slices that includes the at least one missing encoded data slice and wherein the locally decodable redundancy data is stored locally to the processing circuit; and recovering, via the at least one processing circuit, the data based on the at least one rebuilt encoded data slice and the first subset.