Distributed Storage Slice Deletion for Memory-Efficient Decoding
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Current distributed storage and task processing systems face challenges in efficiently managing memory utilization and ensuring data integrity across geographically dispersed storage units, particularly in handling large datasets and complex tasks, while maintaining security and reliability.
Innovation Solution
A distributed computing system that employs dispersed error encoding and decoding schemes, where data is segmented, encoded, and distributed across multiple execution units, allowing for secure and reliable storage and processing of data, with error correction mechanisms to rebuild corrupted slices and ensure data integrity.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If data is distributed across multiple geographically dispersed storage units, then system reliability and fault tolerance are improved, but system complexity and coordination overhead increase
Solution Approach 1:
The patent segments data into multiple slices that are distributed across different storage units. Each slice is independently stored, allowing the system to maintain reliability through distribution while managing complexity through standardized segmentation operations. The data segmentation enables fault tolerance without requiring complex inter-unit coordination for basic storage operations.
Solution Approach 2:
The system changes the parameter of data representation by encoding data in multiple redundant slices with specific mathematical relationships. This parameter transformation allows any sufficient number of slices to reconstruct the original data, providing reliability while the standardized encoding/decoding process manages system complexity through consistent mathematical operations rather than complex coordination protocols.
2Reliability
If error correction encoding is applied to distributed data, then data integrity and fault tolerance are improved, but processing time and computational overhead increase
Solution Approach 1:
The system performs error correction encoding in advance during the data writing phase. By pre-encoding data into multiple slices with built-in redundancy before distribution, the system establishes fault tolerance upfront rather than requiring time-consuming error correction during read operations. This preliminary action reduces processing time during data retrieval while maintaining high fault tolerance.
Solution Approach 2:
The error correction encoding creates redundant copies of data information across multiple slices. Instead of performing complex real-time error correction, the system uses pre-created redundant copies that can be directly used for reconstruction. This copying approach trades increased write time for reduced read time, overall improving the fault tolerance-to-processing-time ratio.
3Reliability
If data is encoded and distributed across multiple storage units, then security and reliability are improved, but data retrieval complexity and access time increase
Solution Approach 1:
The system allows partial retrieval of data by requiring only a sufficient number of slices (k out of n) to reconstruct the original data. This partial action principle enables flexible retrieval where users can access data faster by retrieving from fewer storage units when possible, while still maintaining security and reliability through the minimum threshold requirement. It balances security with reduced retrieval time by not requiring all slices for every access.
Solution Approach 2:
The system changes the parameter of data accessibility by using mathematical encoding that allows reconstruction from any sufficient subset of slices. This parameter transformation maintains security (data cannot be reconstructed from insufficient slices) while enabling faster retrieval (data can be reconstructed from fewer slices than the total distributed). The k-out-of-n parameter structure balances security and retrieval efficiency.
Data Source
AI summary
A method includes encoding data into pluralities of sets of encoded data slices. The method further includes outputting the pluralities of sets of encoded data slices to DST units, wherein each of the DST units stores a slice grouping of encoded data slices. The method further includes dividing the task into a decode threshold number of partial tasks. The method further includes sending a slice deletion policy to the DST units. On a data chunkset by data chunkset basis: the method further includes selecting a decode threshold number of DST units; assigning the decode threshold number of partial tasks to the decode threshold number of DST units; executing the decode threshold number of partial tasks on the slice groupings to produce partial results; and deleting the plurality of sets of encoded data slices of the given data chunkset in accordance with the slice deletion policy.