Dispersed Storage Task Network for Fault-Tolerant Data Processing
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Current distributed computing systems face challenges in efficiently storing and processing large datasets across multiple locations while ensuring data integrity and security, particularly in handling failures and hacking attempts.
Innovation Solution
A distributed computing system that employs dispersed storage and task processing, utilizing error encoding and decoding schemes to encode data into slice groups, which are then processed and stored across multiple geographically dispersed units, allowing for reliable and secure storage and retrieval with tolerance to failures and hacking attempts.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If data is stored across multiple geographically dispersed locations, then data availability and fault tolerance are improved, but system complexity and coordination overhead increase
Solution Approach 1:
The patent segments data into multiple slices that are independently stored across different geographically dispersed locations. Each slice can be independently accessed and recovered, allowing the system to maintain high availability without requiring complex coordinated access protocols across all nodes.
Solution Approach 2:
The patent introduces an intermediary encoding layer that transforms data into encoded slices before distribution. This intermediary step simplifies the storage and retrieval process by enabling independent access to any subset of slices while maintaining data integrity, reducing the coordination complexity among distributed nodes.
2Reliability
If error correction encoding is applied to data, then data integrity and fault tolerance are improved, but processing time and computational resources increase
Solution Approach 1:
The patent applies error correction encoding in advance during the data storage phase, creating redundant encoded slices that can be used for future recovery. This preliminary action ensures that when failures occur, recovery can proceed quickly without requiring complex real-time computation, thus reducing the time loss during fault recovery operations.
3Object-affected harmful factors
If data is encrypted for security, then resistance to hacking attempts is improved, but processing speed and accessibility are reduced
Solution Approach 1:
The patent segments encrypted data into multiple slices distributed across different locations. This segmentation allows parallel processing of individual slices, maintaining processing speed while preserving security. Each slice can be processed independently, avoiding the bottleneck that would occur with monolithic encrypted data processing.
Data Source
Figure 1
Figure 2
Figure 3
AI summary
A method begins by a dispersed storage (DS) processing module selecting a set of distributed storage and task (DST) execution units for executing a task and determining dispersed storage error coding parameters for data. The method continues with the DS processing module dispersed storage error encoding the data in accordance with the parameters to produce a plurality of encoded data blocks and grouping the plurality of encoded data blocks into a plurality of encoded data block groupings. The method continues with the DS processing module partitioning the task into a set of partial tasks, outputting at least some of the plurality of encoded data block groupings to the set of DST execution units, and outputting the set of partial tasks to the set of DST execution units for execution of the set of partial tasks on the at least some of plurality of encoded data block groupings.