Dispersed Storage Data De-duplication via Error Coding Slices
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Conventional data storage systems face challenges with data integrity and security due to the high probability of disc failures in redundant arrays, which increases maintenance demands and introduces security risks from multiple data copies, while also incurring efficiency losses as data volume grows.
Innovation Solution
A distributed storage system that employs error coding dispersal storage functions to partition and encode data into multiple slices, which are then stored across physically diverse locations, allowing for reliable and secure data retrieval even in the event of individual storage device failures, and includes a storage integrity processing unit for periodic verification and rebuilding of corrupted slices.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If multiple redundant disc drives are used to replicate data, then data reliability is improved, but security risks increase due to multiple data copies and maintenance demands increase
Solution Approach 1:
The patent segments data into multiple slices and disperses them across different storage locations. Instead of creating complete redundant copies of entire data sets, the system divides data into fragments and stores them distributedly, thereby maintaining reliability while reducing security risks associated with multiple complete copies
Solution Approach 2:
The patent applies different characteristics to different parts of the stored data. Each slice is stored with unique error correction codes and metadata specific to that slice's location and purpose, allowing localized verification and recovery without exposing the entire data set, thus improving security while maintaining reliability
2Reliability
If multiple redundant disc drives are used to replicate data, then data reliability is improved, but maintenance demands increase
Solution Approach 1:
The patent implements self-healing capabilities through error correction codes embedded with each data slice. When corruption is detected, the system automatically reconstructs affected data using the redundant information distributed across other slices, eliminating the need for manual intervention and reducing maintenance demands while maintaining high reliability
3Reliability
If data is stored in redundant arrays, then data availability is improved, but efficiency losses occur as data volume grows
Solution Approach 1:
By segmenting data into slices and storing them distributedly rather than as complete redundant copies, the system achieves data availability through dispersion rather than duplication. This segmentation allows the storage capacity to scale linearly with added storage resources, maintaining efficiency even as data volume grows
Solution Approach 2:
The patent changes the storage parameter from complete data copies to data slices with error correction overhead. This parameter change optimizes the balance between availability and efficiency, as the overhead remains constant per slice rather than scaling with data volume, allowing efficient storage of large data sets while maintaining high availability
Data Source
Figure 1
Figure 2
Figure 3
AI summary
A method begins with a processing module receiving, from a requesting device, a data storage request that includes data for storage. The method continues with the processing module determining whether substantially identical data is currently stored in a dispersed storage network (DSN) memory. The method continues with the processing module encoding at least a portion of the data using an error coding dispersal storage function to produce a set of encoded data slices, sending the set of encoded data slices to the DSN memory for storage therein, and generating a unique retrieval matrix for the requesting device, wherein the unique retrieval matrix identifies a sub-set of encoded data slices of the set of encoded data slices for subsequent retrieval of the at least a portion of the data, when the substantially identical data is not stored in the DSN memory.