Temporary Sparse Index for Deduplication Storage Memory Optimization
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
The deduplication storage system faces performance issues due to the large size of the single persistent sparse index, which consumes excessive memory and can lead to data corruption and deadlocks during concurrent access, especially as the amount of stored data increases.
Innovation Solution
Generating multiple temporary sparse indexes, each for a single backup process, reduces memory requirements and improves performance by using smaller indexes that are deleted after processing, instead of relying on a single large persistent sparse index.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If a single persistent sparse index is used to track all data units, then the system can maintain complete indexing information, but the index size becomes excessively large consuming too much memory and causing performance degradation
Solution Approach 1:
The patent divides the single persistent sparse index into multiple temporary sparse indexes, each associated with a specific backup process. Each temporary index tracks only the data units relevant to that particular backup process, significantly reducing the memory footprint while maintaining complete indexing for the current operation. After processing, each temporary index is deleted, preventing memory bloat from accumulated historical data.
2Reliability
If a single persistent sparse index is used for concurrent backup processes, then all data units are tracked centrally, but data corruption and deadlocks occur during concurrent access
Solution Approach 1:
The patent assigns a separate temporary sparse index to each backup process, isolating their data tracking operations. This segmentation eliminates concurrent access conflicts to a single index structure, preventing deadlocks and data corruption. Each process operates on its own index independently, simplifying concurrent access management while maintaining data integrity.
Solution Approach 2:
The patent uses temporary sparse indexes that are created for each backup process and automatically deleted after processing completes. These short-lived indexes avoid the persistence and complexity of a single long-lived index structure, eliminating concurrent access issues while maintaining necessary tracking functionality during the backup operation.
3Reliability
If a single large persistent sparse index is used, then comprehensive data tracking is maintained, but processing performance degrades due to the large size of the index
Solution Approach 1:
The patent segments the comprehensive data tracking into multiple smaller temporary indexes, each handling a specific backup process. This segmentation reduces the size of individual index structures that need to be processed, improving processing speed while maintaining complete data tracking for each backup operation through the combined information of all temporary indexes.
Data Source
AI summary
Example implementations relate to deduplication operations in a storage system. An example includes, in response to initiation of a new backup process to store a first stream of data, initializing a temporary sparse index to be stored in a memory of a deduplication storage system; identifying a cloned portion of the first data stream; identifying at least one container index associated with the cloned portion of the first data stream; identifying a set of hook points included in the at least one container index; and populating the temporary sparse index with a set of entries, the set of entries mapping the identified set of hook points to the at least one container index.