Mapping Slice Groupings in Dispersed Storage Networks
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Current dispersed storage networks face challenges in securely and reliably storing and retrieving large amounts of data across multiple geographically distributed locations, particularly in ensuring data integrity and availability in the face of device and network failures.
Innovation Solution
A distributed computing system that employs dispersed error encoding and decoding techniques, where data is segmented, encoded, and distributed across multiple storage units, allowing for secure storage and retrieval while being resilient to failures through error correction mechanisms and secure encryption.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If data is distributed across multiple geographically dispersed storage units, then data availability and fault tolerance are improved, but system complexity and difficulty of managing data integrity increase
Solution Approach 1:
The patent segments data into multiple slices and distributes them across different storage units. Each slice is independently stored, allowing the system to maintain data availability even when some storage units fail. This segmentation approach directly addresses the contradiction by improving reliability through distribution while managing complexity through systematic data division.
Solution Approach 2:
The patent introduces mapping structures as intermediaries that manage the relationship between data slices and storage units. These mapping structures provide a layer of abstraction that simplifies the management of distributed data, reducing the complexity burden while maintaining the reliability benefits of geographic distribution.
2Reliability
If error correction encoding is applied to distributed data, then data integrity is improved, but processing time and computational overhead increase
Solution Approach 1:
The patent applies error correction encoding to data slices during the initial storage process rather than during retrieval. This preliminary action ensures data integrity is established upfront, allowing faster retrieval operations without the computational overhead of real-time error correction, thus addressing the time penalty concern.
Solution Approach 2:
The patent creates redundant copies of data through error correction encoding, where additional encoded slices are stored alongside original data slices. This copying approach provides fault tolerance and integrity verification without requiring complex real-time processing during data access, as the redundancy is pre-computed and stored.
3Object-affected harmful factors
If data is encrypted for secure storage, then security is improved, but retrieval speed and processing efficiency decrease
Solution Approach 1:
The patent applies encryption selectively to specific portions of data or uses different encryption schemes for different data slices. This local quality approach allows the system to maintain security while optimizing retrieval speed for non-sensitive or frequently accessed data, balancing security requirements with productivity concerns.
Data Source
AI summary
A distributed storage network includes a method of receiving data and a corresponding task, selecting one or more storage units for the task based on a capability level associated with each of the storage units, identifying a plurality of data groups of the data, determining a chunk size based on the plurality of data groups, determining processing parameters of the data based on the chunk size, generating a set of chunksets from the plurality of data groups in accordance with the chunk size and processing parameters and encoding the set of chunksets in accordance with the processing parameters to produce slice groupings. The method further includes determining task partitioning based on the storage units and the processing parameters, partitioning the task based on the task partitioning to produce partial tasks and sending the slice groupings and corresponding partial tasks to the storage units.