Dispersed Streaming Data Storage Using Error-Coded Slices
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Conventional data storage systems face issues with data integrity and security due to the failure of physical movement-based memory devices, such as disc drives, which can lead to bit-level corruption and complete failure within three years, and redundant array of independent discs (RAID) solutions increase maintenance demands and security risks with multiple copies of data.
Innovation Solution
A distributed storage network (DSN) system that uses error-coded data slices stored across multiple physically diverse locations, managed by a DS processing unit and DS managing unit, ensuring data integrity and security through error correction and dispersal algorithms, allowing for reliable and secure data storage and retrieval despite device failures.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If multiple copies of data are stored using RAID, then data reliability is improved, but security risk and maintenance demand increase
Solution Approach 1:
The data is divided into multiple segments and distributed across different storage locations. Each segment is a necessary component for reconstructing the original data, so that no single location contains a complete copy. This segmentation approach improves security by eliminating centralized data copies while maintaining reliability through distributed reconstruction capability.
Solution Approach 2:
Error correction codes act as an intermediary mechanism between the stored segments and data reconstruction. These codes enable the system to tolerate segment losses or corruptions without requiring exact duplicate copies, thereby achieving reliability through intelligent mediation rather than simple replication.
2Reliability
If multiple copies of data are stored using RAID, then data reliability is improved, but maintenance demand increases
Solution Approach 1:
By segmenting data into distributed pieces with error correction codes, the system eliminates the need to manage and synchronize multiple complete data copies. Maintenance operations only need to handle individual segments rather than coordinating across multiple RAID arrays, significantly reducing maintenance complexity and demand.
3Object-affected harmful factors
If data is distributed across multiple locations, then security is improved, but system complexity increases
Solution Approach 1:
The error correction code mechanism serves multiple functions simultaneously: it enables data reconstruction from partial segments, provides tolerance for segment losses, and facilitates secure distribution across locations. This multi-functionality reduces the need for separate security mechanisms, thereby managing complexity while achieving security goals.
Data Source
AI summary
A method begins by a processing module determining properties of streaming data. The method continues with the processing module determining required dispersed storage resources based on the properties of the streaming data, identifying a plurality of sets of dispersed storage resources, and establishing first error coding dispersed storage function parameters for a first set of dispersed storage resources of the plurality of sets of dispersed storage resources when the required dispersed storage resources requires the plurality of sets of dispersed storage resources. In addition, the method continues with the processing module enabling partitioning of the streaming data into a plurality of data streams and enabling routing of a first data stream of the plurality of data streams to the first set of dispersed storage resources, wherein the first set of dispersed storage resources converts the first data stream into pluralities of error coded data slices.