Dispersed Storage Network Data Deduplication via Object Linking
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Current distributed storage and task processing systems face challenges in efficiently storing and retrieving large amounts of data across geographically dispersed locations while ensuring data integrity and security, particularly in the presence of failures and network complexities.
Innovation Solution
A distributed computing system that employs dispersed error encoding and decoding techniques to segment and distribute data across multiple storage units, allowing for secure storage and retrieval of data, as well as the execution of tasks on dispersed data sets, using a network of geographically different DST execution units.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If data is distributed across multiple geographically dispersed storage units, then data availability and security are improved, but system complexity and failure detection difficulty increase
Solution Approach 1:
The patent segments data into multiple slices that are independently stored across different geographically dispersed storage units. Each slice is a complete but redundant representation of the original data, allowing any sufficient number of slices to reconstruct the full data set. This segmentation enables data to be distributed without increasing operational complexity, as each storage unit handles identical data representations.
Solution Approach 2:
The patent transforms data from a single monolithic form into multiple parameter variations (slices with different redundancy levels). By encoding data with error correction codes that create multiple valid representations, the system changes the parameter of data representation rather than creating fundamentally different storage structures, simplifying the management of distributed storage units.
2Reliability
If data is segmented and distributed across multiple storage units, then data security and fault tolerance are improved, but data retrieval complexity increases
Solution Approach 1:
The patent performs preliminary error correction encoding and slice creation before data distribution. Storage units receive pre-configured slices with embedded redundancy information, so that during retrieval, the system only needs to collect sufficient slices and apply predetermined decoding algorithms. This preliminary preparation eliminates complex real-time decision-making during data retrieval operations.
Solution Approach 2:
The patent creates multiple complete copies of data in sliced form across different storage units. Each slice contains sufficient information to reconstruct the original data when combined with other slices. This copying approach simplifies retrieval because any sufficient combination of slices can be used, eliminating the need for complex hierarchical or sequential access patterns.
3Reliability
If error correction encoding is applied to dispersed data, then data integrity is improved, but processing time and computational resources increase
Solution Approach 1:
The patent segments the error correction encoding process into parallel operations applied to individual data slices during the initial storage phase. Once slices are created and distributed, verification and reconstruction use efficient algebraic decoding algorithms that operate independently on each slice, minimizing processing time during retrieval while maintaining comprehensive data integrity checks.
Solution Approach 2:
The patent transforms error correction from a continuous verification process into discrete parameter-based validation. By encoding data with fixed redundancy parameters and using corresponding decoding algorithms, the system changes error correction from an ongoing computational burden to a structured process with predictable, minimized processing requirements.
Data Source
AI summary
A method begins by a dispersed storage (DS) processing module receiving data for storage and generating a dispersed storage network (DSN) source name for the data. The method continues with the DS processing module determining whether substantially identical data to the data has been previously stored in memory of the DSN. When the substantially identical data has been previously stored in the memory of the DSN, the method continues with the DS processing module generating an object linking file that links the data to the substantially identical data, dispersed storage error encoding the object linking file to produce a set of encoded link file slices, and outputting the set of encoded link file slices for storage in the memory of the DSN.