Dispersed Storage Encoding for Reliable Streaming Data Slices
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Current distributed computing systems face challenges in securely and reliably storing and retrieving large amounts of data across multiple geographically dispersed locations while ensuring data integrity and availability, especially in the presence of failures and network issues.
Innovation Solution
A distributed computing system that employs dispersed error encoding and decoding techniques to segment and distribute data across multiple storage units, allowing for secure storage and retrieval of data even in the event of failures, using a network of distributed storage and task processing units that encode data into slice groups and execute tasks on these encoded data slices across multiple execution units.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If data is distributed across multiple geographically dispersed storage units, then data availability and fault tolerance are improved, but system complexity and coordination overhead increase
Solution Approach 1:
The patent segments data into multiple slices that are independently stored across different geographically dispersed storage units. Each slice is a self-contained unit that can be retrieved independently, eliminating the need for complex coordination between storage units while maintaining data availability. The segmentation is achieved through error encoding that divides original data into multiple independent slices.
Solution Approach 2:
The patent creates multiple copies of encoded data slices and distributes them across different storage units. Rather than storing redundant copies of entire datasets, the system stores distributed copies of encoded slices, where any sufficient combination of slices can reconstruct the original data. This copying approach improves availability without proportionally increasing system complexity.
2Reliability
If error correction encoding is applied to data, then data integrity is improved, but processing time and computational resources increase
Solution Approach 1:
The patent applies error correction encoding in advance during the data storage phase, transforming original data into multiple encoded slices before distribution. This preliminary encoding action ensures that when data is retrieved, no additional decoding or error correction processing is needed at read time, as the encoded slices are self-contained and can be directly used for reconstruction.
Solution Approach 2:
The error correction encoding process segments data into independent slices that can be processed and retrieved separately. This segmentation allows parallel processing during encoding and enables selective retrieval of only the necessary number of slices for reconstruction, reducing overall processing time compared to traditional error correction methods that require processing entire datasets.
3Reliability
If data is segmented into multiple slices and distributed, then fault tolerance is improved, but data retrieval complexity increases
Solution Approach 1:
The patent changes the parameter of data representation by transforming original data into encoded slices with specific mathematical relationships. Each slice contains encoded information that, when combined with a sufficient number of other slices, can reconstruct the original data. This parameter transformation enables simple retrieval operations where any sufficient combination of slices works, eliminating complex retrieval logic.
Solution Approach 2:
The system creates distributed copies of encoded data slices across multiple storage units. Each copy is identical in content but stored independently, allowing retrieval operations to access any available copy without complex coordination. The copying mechanism simplifies retrieval by allowing clients to request slices from any storage unit in the distributed network.
4Reliability
If dispersed storage is implemented across multiple locations, then security is improved, but network bandwidth requirements increase
Solution Approach 1:
The patent segments data into multiple smaller slices that are distributed across different storage locations. This segmentation reduces the network bandwidth required for any single transfer operation, as each slice is smaller than the complete dataset. The segmented approach also enables parallel transfer of multiple slices simultaneously, improving security through distribution while managing network bandwidth requirements.
Solution Approach 2:
The system implements partial replication of data by storing encoded slices rather than complete copies of the original data. Each storage location holds only a portion of the encoded information, which is sufficient for reconstruction when combined with other slices but insufficient alone. This partial storage approach reduces total network bandwidth requirements compared to full replication while maintaining security through distribution.
Data Source
AI summary
A method includes receiving streaming data. The method further includes dividing the streaming data into data segments. The method further includes performing a unity matrix function on a data segment to produce a set of encoded data slices. The method further includes storing the set of encoded data slices in a set of storage units of the DSN. After storage of the set of encoded data slices, the method further includes performing a redundancy portion of an encoding matrix on the set of encoded data slices to produce a set of redundancy encoded data slices and storing the set of redundancy encoded data slices in a set of set of storage units of the DSN.