Dispersed Storage Network Data Slicing and Encoding
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Current distributed storage and task processing systems face challenges in ensuring data integrity and security, particularly in handling large datasets and complex tasks across multiple geographically dispersed storage units, where data loss and unauthorized access are significant concerns.
Innovation Solution
A distributed computing system that employs dispersed error encoding and decoding, allowing data to be encoded into multiple slices stored across different locations, with a decentralized agreement module for secure storage and task execution, and a decentralized agreement module for resource selection to enhance data integrity and security.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If data is stored in a dispersed storage network across multiple geographically dispersed units, then data security and fault tolerance are improved, but system complexity and coordination overhead increase
Solution Approach 1:
The patent segments data into multiple slices and disperses them across different storage units in the network. Each slice is independently stored, and the system uses slicing algorithms to divide and reconstruct data, thereby improving reliability while managing complexity through structured segmentation
Solution Approach 2:
The patent introduces intermediary components including a slicing algorithm, encoding/decoding modules, and coordination protocols that mediate between data storage and retrieval operations. These intermediaries abstract the complexity of dispersed storage while maintaining data integrity across distributed units
2Reliability
If data is encoded into multiple slices and stored across different locations, then fault tolerance against multiple failures is improved, but storage overhead and retrieval complexity increase
Solution Approach 1:
The patent applies preliminary encoding and slicing actions during data ingestion, transforming data into a fault-tolerant format before storage. This preliminary processing ensures that even if multiple storage units fail, the data can be reconstructed from remaining slices, while the encoding structure manages overhead through efficient algorithms
3Reliability
If a decentralized agreement module is used for secure storage and task execution, then data security is improved, but processing time and coordination overhead increase
Solution Approach 1:
The patent implements partial decentralized agreement mechanisms where not all storage units need to reach consensus for every operation. Instead, a threshold number of units participate in agreement protocols, providing sufficient security while reducing coordination time and processing delays
Data Source
AI summary
A method begins with a set of storage units receiving a plurality of sets of non-locking write requests from a plurality of computing devices. The method continues with each storage unit storing an encoded data slice of a respective one of the non-locking write requests of each set and sending a write response regarding the respective one of the non-locking write requests. The method continues with a computing device determining whether a threshold number of write responses regarding a corresponding one of the plurality of sets of non-locking write requests has an expected ordering indication. When the threshold number of write responses has the expected ordering indication, the method continues with the computing device sending a set of write finalize requests to the set of storage units to facilitate finalizing storing the set of encoded data slices of the corresponding one of the plurality of sets of non-locking write requests.