Distributed Storage Memory Release Using Slice Retrieval Thresholds
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Current distributed storage and task processing systems face challenges in efficiently managing and retrieving large amounts of data across multiple devices while ensuring data integrity and security, particularly in scenarios where data is distributed across geographically disparate locations.
Innovation Solution
A distributed computing system that employs dispersed error encoding and decoding techniques to segment and store data across multiple execution units, allowing for reliable and secure data retrieval and task processing, even in the presence of failures, by using a network of distributed storage and task processing units that mimic conventional operating system interfaces for communication.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If data is stored across multiple distributed storage devices, then data availability and fault tolerance are improved, but data integrity and security become more difficult to ensure
Solution Approach 1:
The patent segments data into multiple slices and distributes them across different storage devices. Each slice is independently stored, allowing the system to maintain data availability even when some devices fail. This segmentation approach directly addresses the contradiction by enabling fault tolerance while managing integrity through controlled distribution.
Solution Approach 2:
The patent applies error correction encoding to data slices before distribution across storage devices. This preliminary encoding ensures that even if some slices are lost or corrupted during transmission or storage, the original data can be reconstructed. This pre-processing step resolves the contradiction by establishing data integrity mechanisms in advance.
2Reliability
If error correction encoding is applied to data, then data integrity is improved, but processing time and computational resources increase
Solution Approach 1:
The patent applies error correction encoding selectively to critical data portions rather than uniformly to all data. By identifying and prioritizing which data slices require robust error correction, the system maintains data integrity for essential information while reducing overall processing time and computational overhead.
Solution Approach 2:
The patent dynamically adjusts error correction parameters based on data characteristics and storage conditions. By changing encoding strength, slice size, and distribution patterns according to specific requirements, the system optimizes the balance between data integrity and processing efficiency for different types of data and operational contexts.
Data Source
AI summary
A method begins by a distributed storage (DS) processing module retrieving a data slice from a local memory and performing a partial task on the data slice. When the performing of the partial task is complete, the method continues with the DS processing module determining whether at least a retrieval number of slices of a set of slices of a data segment that includes the data slice is available from a set of DST execution units. When the at least a retrieval number of slices is available, the method continues with the DS processing module deleting the data slice from the local memory. When the at least a retrieval number of slices of the set of slices is not available, the method continues with the DS processing module determining whether execution of a task on the data segment is complete and deleting the data slice when the execution is complete.