Dispersed Storage Task Partitioning Without Redundant Copies
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing data storage and processing systems face challenges in efficiently managing large volumes of data across distributed networks, particularly in ensuring data integrity and security while handling complex tasks, and are vulnerable to failures without redundant copies.
Innovation Solution
A distributed computing system that employs dispersed storage and task processing, utilizing error encoding and decoding techniques to store and process data across geographically diverse sites, ensuring data integrity and security, and allowing for fault tolerance without redundant copies.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Quantity of substance
If data is stored in a distributed network without redundant copies, then storage efficiency is improved, but reliability deteriorates due to vulnerability to failures
Solution Approach 1:
The patent segments data into multiple data blocks and disperses them across different storage devices in the network. Each data block can be independently stored and retrieved, allowing the system to achieve both storage efficiency through distributed placement and reliability through the ability to reconstruct data from multiple sources.
Solution Approach 2:
The patent creates encoded copies of data blocks using error correction encoding. These encoded copies are distributed across the network and can be used to reconstruct original data even if some copies are lost or corrupted, thereby maintaining reliability without requiring identical redundant copies.
2Reliability
If error correction encoding is applied to data blocks, then reliability is improved, but processing complexity increases
Solution Approach 1:
The patent applies error correction encoding to data blocks before they are dispersed across the network. This preliminary encoding action ensures that data integrity is protected from the outset, and the decoding process during retrieval is simplified because the encoding structure is already in place.
Solution Approach 2:
The patent transforms data blocks by applying error correction encoding, which changes the parameter structure of the data. This transformation enables reliable data recovery through mathematical relationships established by the encoding scheme, managing complexity through structured parameter changes rather than complex recovery algorithms.
3Reliability
If data blocks are dispersed across geographically diverse sites, then security is improved, but system complexity increases
Solution Approach 1:
The patent divides data into multiple blocks and disperses them across geographically diverse storage devices. This segmentation provides security because no single location contains the complete data, yet the system remains manageable through automated encoding and decoding processes that handle the distribution complexity.
Data Source
AI summary
A method for execution by a processing module of a storage network includes determining processing parameters for data based on a number of storage and execution units of the storage network to be utilized in processing the data, where the data is associated with a task. The method further includes task partitioning of the task based on the number of storage and execution units and the processing parameters. The method further includes processing the data in accordance with the processing parameters to produce slice groupings. The method further includes partitioning the task based on the task partitioning to produce partial tasks. The method further includes sending the slice groupings and corresponding partial tasks to the storage and execution units.