Distributed Data Stream Decoding with Threshold Coded Slices
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Current computer storage systems face challenges with data integrity and security due to the failure of memory devices, particularly those using physical movement technologies, which can lead to bit-level corruption and complete failure within three years, and redundant array of independent discs (RAID) solutions increase maintenance demands and security risks with multiple copies of data.
Innovation Solution
A distributed storage system that employs error coding dispersal storage to partition data into slices, which are then stored across multiple physically diverse locations, allowing for reliable and secure data retrieval even in the event of device failures, using a network of user devices, distributed storage processing units, and storage integrity processing units to manage and verify data integrity.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If RAID solutions are used to protect against device failures, then data reliability is improved, but maintenance demands and security risks increase
Solution Approach 1:
The patent segments data into multiple slices and distributes them across different storage devices using error coding (e.g., Reed-Solomon codes). Instead of creating full redundant copies like RAID, the system divides data into n slices where any k slices can reconstruct the original data. This reduces the number of copies needed while maintaining reliability, thereby lowering maintenance demands.
Solution Approach 2:
The system changes the redundancy parameter from fixed copy-based redundancy (RAID) to flexible slice-based redundancy with configurable thresholds. By adjusting the error coding parameters (n, k pairs), the system can optimize between storage capacity and reliability without increasing maintenance complexity, as the same distributed architecture handles all recovery operations uniformly.
2Reliability
If multiple copies of data are stored for redundancy, then data reliability is improved, but security risks increase
Solution Approach 1:
By segmenting data into slices and distributing them across multiple devices, the system ensures that no single device contains a complete copy of the data. This segmentation reduces security risks because compromising one device only exposes partial information, making it harder for attackers to retrieve complete data even if they gain access to multiple devices.
Solution Approach 2:
Different slices of data can be stored with different security characteristics or access permissions. The system can apply local quality control where specific slices have enhanced protection or are stored in different security zones, allowing flexible security management that reduces overall risk while maintaining reliability through the distributed nature of storage.
3Reliability
If data is distributed across multiple locations, then security and reliability are improved, but storage capacity efficiency decreases
Solution Approach 1:
The system uses error coding parameters (n, k) to control the trade-off between distributed storage and capacity efficiency. By adjusting these parameters, the system can achieve high reliability with minimal redundancy overhead. For example, with (5,3) coding, only 33% redundant capacity is needed to tolerate 2 failures, which is more efficient than traditional RAID configurations while maintaining distributed storage benefits.
Data Source
AI summary
A method begins by a dispersed storage (DS) processing module receiving one or more pairs of coded values and continues with the DS processing module creating a received coded matrix from the one or more pairs of coded values, followed by the generation of a data matrix. When a decode threshold number of pairs of coded values are included in a first plurality of data blocks corresponding to a data stream from the data matrix and the decode threshold number of pairs of coded values are determined to be valid, the decode threshold number of pairs of coded values are decoded. The method continues with the generation of a second data matrix and the data stream is extracted from the second data matrix.