Dispersed Storage Task Allocation Using Encoded Data Slices
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing distributed storage networks face challenges in ensuring data integrity and security, particularly in handling large volumes of data across multiple storage units, where data loss or corruption can occur due to storage unit failures without adequate redundancy or backup systems.
Innovation Solution
A dispersed storage network (DSN) architecture that employs error encoding using Cauchy Reed-Solomon encoding to distribute data across multiple storage units, allowing for data recovery even with a significant number of storage unit failures, and includes a managing unit for secure data storage and task processing, with features like user profile management and billing, and an integrity processing unit for rebuilding corrupted data slices.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If data is stored in a distributed manner across multiple storage units, then data availability and fault tolerance are improved, but data integrity and security become harder to ensure due to increased exposure to storage unit failures and corruption
Solution Approach 1:
The patent segments data into multiple slices and distributes them across different storage units. Each slice is independently stored, allowing the system to tolerate failures of individual storage units while maintaining overall data availability and integrity through the use of error correction codes.
Solution Approach 2:
The patent implements error correction coding and redundancy mechanisms in advance before data is stored. This allows the system to recover from storage unit failures and data corruption without losing the original data, effectively cushioning against harmful factors before they can cause permanent damage.
2Reliability
If error correction encoding is applied to protect data, then data integrity is improved, but processing time and computational resources increase
Solution Approach 1:
The patent performs error correction encoding in advance during the data writing phase, rather than waiting until the reading phase. This preliminary action ensures data integrity is established before storage, reducing the processing time required during data retrieval operations.
Solution Approach 2:
The patent divides data into smaller slices and applies error correction encoding to each slice independently. This segmentation allows for parallel processing of encoding operations, reducing the overall processing time while maintaining comprehensive data integrity protection.
3Reliability
If data is dispersed across multiple storage units, then fault tolerance is improved, but system complexity increases due to coordination and management requirements
Solution Approach 1:
The patent implements a universal error correction code that can be applied to any data slice regardless of its location or content. This multi-functional approach simplifies system management by providing a consistent method for protecting all data across diverse storage units, reducing the complexity of coordinating fault tolerance across the distributed system.
Data Source
AI summary
A processing system in a dispersed storage and a task (DST) network operates by receiving data and a corresponding task; identifying candidate DST execution units for executing partial tasks of the corresponding task; receiving distributed computing capabilities of the candidate DST execution units; selecting a subset of DST execution units of the candidate DST execution units to favorably execute the partial tasks of the corresponding task; determining task partitioning of the corresponding task into the partial tasks based on one or more of the distributed computing capabilities of the subset of DST execution units; determining processing parameters of the data based on the task partitioning; partitioning the tasks based on the task partitioning to produce the partial tasks; processing the data in accordance with the processing parameters to produce slice groupings; and sending the slice groupings and the partial tasks to the subset of DST execution units.