Cooperative Data Ingestion for Dispersed Slice Storage Integrity
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Current cloud data storage systems face challenges in ensuring data integrity and availability across distributed storage networks, particularly in scenarios where multiple storage units fail, as they often require redundant copies and are vulnerable to data loss and unauthorized access.
Innovation Solution
A dispersed storage network (DSN) utilizing Cauchy Reed-Solomon error encoding and decoding processes, where data is segmented into encoded slices stored across multiple geographically diverse sites, allowing for error correction and secure storage without the need for redundant copies, with a managing unit coordinating storage and integrity processing units to maintain data integrity and security.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If redundant copies of data are stored across distributed storage units, then data availability is improved, but storage space is wasted and system complexity increases
Solution Approach 1:
The patent segments data into multiple slices and disperses them across different storage units in a distributed network. Instead of storing complete redundant copies, the system divides data into fragments that can be reconstructed from a subset of slices, thereby reducing total storage requirements while maintaining data availability.
Solution Approach 2:
The system changes the storage parameter from storing complete data copies to storing encoded data slices. By applying error correction encoding, the patent transforms the data representation such that fewer physical slices are needed to reconstruct the original data, optimizing the balance between storage efficiency and data availability.
2Reliability
If multiple redundant copies of data are maintained, then data availability is improved, but vulnerability to unauthorized access and data loss increases
Solution Approach 1:
By segmenting data into multiple encrypted slices distributed across different storage units, the patent reduces the risk of unauthorized access. An attacker would need to compromise multiple distributed storage units simultaneously to reconstruct the original data, significantly increasing the security threshold compared to targeting single redundant copies.
Solution Approach 2:
The patent introduces error correction codes as an intermediary layer between the stored slices and the reconstructed data. This intermediary mechanism not only enables data recovery from lost slices but also adds a layer of security by requiring valid error correction verification to successfully reconstruct original data, preventing unauthorized reconstruction attempts.
3Reliability
If data is dispersed across multiple geographically diverse sites, then fault tolerance is improved, but data retrieval speed deteriorates
Solution Approach 1:
The patent implements preliminary action by pre-caching frequently accessed data slices in geographically distributed edge locations closer to users. This allows the system to maintain geographically diverse storage for fault tolerance while serving common data retrieval requests from nearby cached copies, thereby reducing retrieval latency without sacrificing distributed storage benefits.
Data Source
AI summary
A method begins with receiving data objects. As the data objects are received, the method continues by assigning a data type identifier to each data object based on the data object being a data type. The method continues with sending data objects having a first data type identifier to a first group of computing devices and sending data objects having a second data type identifier to a second group. The method continues with interpreting, by the first group, the data objects having the first data type identifier to sort, based on sorting criteria the data objects into a first processing category and a second processing category. The method continues with dispersed storage error encoding the data objects in the second processing category based on short term storage error encoding parameters to produce pluralities of sets of encoded data slices and sending the slices to storage and execution units.