Concatenated Data Encoding for Reliable Dispersed Storage
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Current distributed storage and task processing systems face challenges in efficiently managing and processing large datasets across multiple devices, particularly in ensuring data integrity and security, and in efficiently executing tasks on dispersed data without loss or corruption.
Innovation Solution
A distributed computing system that employs dispersed error encoding and decoding techniques to store and process data across multiple geographically dispersed execution units, utilizing error correction schemes to ensure data integrity and security, and a task distribution module to manage task execution across these units.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If data is stored using traditional distributed storage systems, then data can be stored across multiple devices, but data integrity and security cannot be ensured without redundant copies
Solution Approach 1:
The patent segments data into multiple slices and disperses them across different storage devices. Each slice is encoded using error correction codes, allowing the system to reconstruct the original data from any sufficient subset of slices without requiring complete redundancy. This segmentation approach enables reliable data storage while minimizing the quantity of redundant copies needed.
Solution Approach 2:
The system changes the parameter of data representation by applying error correction encoding transformations. Instead of storing raw data copies, the data is transformed into encoded slices that can be mathematically reconstructed. This parameter change allows the system to achieve data integrity with fewer physical copies, resolving the contradiction between reliability and quantity of redundant copies.
2Productivity
If large datasets are processed across multiple dispersed execution units, then processing capacity increases, but data loss or corruption may occur during transmission and processing
Solution Approach 1:
The system performs preliminary error correction encoding before data is dispersed to multiple execution units. This preliminary action embeds redundancy and correction capabilities within the data slices themselves, protecting against data loss or corruption during transmission and processing. The encoding is done in advance, allowing the system to maintain high processing capacity while ensuring data integrity through pre-established error correction mechanisms.
3Reliability
If data is dispersed across multiple geographically distributed storage units, then system availability and fault tolerance improve, but data reconstruction becomes more complex
Solution Approach 1:
The patent uses mathematical copying through error correction codes to create redundant information within encoded slices. Instead of physically copying data across multiple locations, the system creates mathematical representations that can be reconstructed from any sufficient subset of slices. This approach improves fault tolerance while keeping reconstruction complexity manageable through standardized decoding algorithms.
Data Source
AI summary
A method begins by a processing module of a dispersed storage network (DSN) concatenating a plurality of independent data objects into a concatenated data object and performing a dispersed storage error encoding function on the concatenated data object to produce a set of data-based encoded data slices and a set of redundancy-based encoded data slices. The method continues with the processing module outputting the set of data-based encoded data slices to a first set of storage units for storage and outputting the set of redundancy-based encoded data slices to a second set of storage units for storage.