Distributed Storage Mapping with Encoded Slices for Data Integrity
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Current distributed storage and task processing systems face challenges in securely and reliably storing and retrieving large amounts of data across multiple locations while ensuring data integrity and supporting complex tasks, particularly in the presence of failures and network issues.
Innovation Solution
A distributed computing system that employs dispersed error encoding and decoding, where data is segmented, encoded, and distributed across multiple storage units, allowing for reliable storage and retrieval, and supports task execution by partitioning tasks into partial tasks executed by geographically diverse units, ensuring data integrity and security through error correction and secure encoding.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If data is distributed across multiple storage units, then reliability is improved, but device complexity increases
Solution Approach 1:
The patent segments data into multiple data slices and distributes them across different storage units. Each slice is independently stored, and the system uses segmentation identifiers to track and retrieve specific portions of data. This segmentation approach improves reliability by ensuring that failure of one storage unit does not result in complete data loss.
Solution Approach 2:
The patent introduces an intermediary encoding layer that transforms original data into encoded data slices before distribution. This intermediary process includes error correction coding and encryption, which simplifies the complexity of direct data distribution while maintaining high reliability through built-in error handling and security mechanisms.
2Reliability
If error correction encoding is applied, then data integrity is improved, but processing time increases
Solution Approach 1:
The patent applies error correction encoding and encryption as preliminary actions during the data storage phase. By pre-processing data with error correction codes and cryptographic transformations before distribution, the system ensures that data integrity is maintained without requiring additional processing time during retrieval operations, as the protective mechanisms are already in place.
Solution Approach 2:
The patent creates multiple copies of encoded data slices across different storage units, where each copy contains redundant error correction information. This copying strategy allows the system to retrieve data from any sufficient number of slices without requiring all original slices, thereby reducing processing time while maintaining data integrity through the redundant encoded copies.
3Object-affected harmful factors
If data is encrypted, then security is improved, but processing speed decreases
Solution Approach 1:
The patent divides encrypted data into smaller segments or slices, each independently encrypted with the same cryptographic algorithm. This segmentation allows parallel processing of multiple encrypted slices during storage and retrieval operations, thereby maintaining security while improving overall processing speed through concurrent operations on divided data portions.
Data Source
AI summary
A method begins by a dispersed storage (DS) processing module mapping a set of data partitions to a set of storage regions. For each data partition, the method continues with the DS processing module segmenting the data partition into a plurality of data segments and designating a first data segment. The method continues with the DS processing module generating data storage mapping information. The method continues with the DS processing module encoding the data storage mapping information to produce at least one set of encoded mapping information slices and for each data partition, encoding the plurality of data segments to produce a plurality of sets of encoded data slices. The method continues with the DS processing module outputting the at least one set of encoded mapping information slices and, for each data partition, the plurality of sets of encoded data slices to the DSN for storage therein.