Nested Dispersed Storage with Sub-Slicing for Fault-Tolerant Data
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing data storage solutions in computing systems face challenges in providing continuous data availability, minimizing the impact of multiple memory element failures, ensuring data security, and compatibility with various storage standards, particularly in scenarios where disk drives fail and data integrity is compromised.
Innovation Solution
A dispersed storage network (DSN) system that breaks down data into error-coded slices, disperses them across multiple storage units, and allows for sub-slicing to enhance redundancy and security, using error encoding functions and sub-slicing algorithms to ensure data integrity and availability across multiple layers of storage.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If RAID replicates data into two or more copies to reduce disk drive failure risk, then data reliability is improved, but storage overhead increases significantly (e.g., RAID 5 adds 50% overhead, RAID 6 requires minimum 4 disks)
Solution Approach 1:
The patent segments data into multiple slices and disperses them across multiple storage units using error-correcting codes. Instead of creating full copies like RAID, the system divides data into N slices where only K slices are needed for reconstruction, achieving reliability with less storage overhead. For example, with (N,K) = (5,3), 5 storage units hold data that can be reconstructed from any 3 of them, using only 60% overhead compared to RAID 5's 50% but with better distributed failure protection.
Solution Approach 2:
The patent changes the fundamental parameter of data protection from replication (RAID) to erasure coding with configurable (N,K) parameters. This allows flexible adjustment of the trade-off between reliability and storage overhead by selecting different N and K values, enabling systems to optimize based on specific reliability requirements rather than being locked into fixed RAID configurations.
2Quantity of substance
If more disks are added to RAID to increase storage capacity, then storage capacity is improved, but the probability of disk failure increases
Solution Approach 1:
The patent segments data across N storage units with the ability to tolerate up to N-K simultaneous failures. This segmentation approach allows the system to scale storage capacity by adding more units while maintaining constant reliability characteristics, unlike RAID where adding disks increases failure probability beyond the protection threshold.
Solution Approach 2:
The patent implements beforehand cushioning by pre-distributing error-correcting codes across all storage units during data writing. This prepares the system in advance to handle any combination of up to N-K failures without requiring real-time detection or special recovery procedures, cushioning against the increased failure probability that comes with scaling.
3Reliability
If RAID mirroring is implemented at different physical locations to reduce data loss risk, then data reliability is improved, but vulnerability to unauthorized access increases
Solution Approach 1:
The patent segments data into encrypted slices distributed across multiple storage units, where each slice alone is useless without the corresponding error-correcting code portions from other units. This segmentation provides both reliability through distributed storage and security through cryptographic slicing, addressing both concerns simultaneously rather than the security-reliability tradeoff in RAID mirroring.
Solution Approach 2:
The patent introduces error-correcting codes as an intermediary layer between the data and storage units. This intermediary not only enables reconstruction from partial data but also acts as a security mechanism, as the coded slices cannot be meaningfully interpreted without the complete set, providing both fault tolerance and access control.
Data Source
AI summary
A method begins by a DS processing module generating a plurality of encoded slices from a data segment using an error encoding function. The method continues with the DS processing module identifying a plurality of DS storage units for storing the plurality of encoded slices. The method continues with the DS processing module selecting an encoded slice of the plurality of encoded slices for sub-slicing using a sub-slicing encoding function to produce a selected encoded slice. The method continues with the DS processing module outputting the plurality of encoded slices to the plurality of DS storage units. The method continues with the DS processing module outputting a command to a DS storage unit of the plurality of DS storage units corresponding to the selected encoded slice, wherein the command includes an instruction to sub-slice the selected encoded slice.