Dynamic Deduplication for Backup Data with Poor Locality
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Conventional deduplication processes face challenges in efficiently handling backup data sets that exhibit poor locality, where data is not consistently stored due to techniques like multiplexing, leading to inefficiencies in identifying and removing redundant data.
Innovation Solution
A method and system that detect the type of data in backup sets, calculate comparison metrics and locality metrics, and adjust deduplication parameters to handle data patterns effectively, using summaries and moving averages to optimize the deduplication process.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Quantity of substance
If conventional deduplication processes are used on backup data sets with poor locality, then storage space is reduced, but deduplication efficiency and performance deteriorate
Solution Approach 1:
The system performs preliminary analysis of backup data sets to detect poor locality patterns before executing the full deduplication process. By identifying multiplexed data patterns in advance, the system can prepare appropriate handling strategies, preventing performance degradation during the actual deduplication operation
Solution Approach 2:
The deduplication system dynamically adapts its behavior based on detected data patterns. When poor locality is detected, the system adjusts its processing approach to handle multiplexed data appropriately, switching between different processing modes to maintain efficiency while still achieving storage reduction
2Speed
If multiplexing is used to speed up backup processes, then backup speed increases, but data locality deteriorates
Solution Approach 1:
The system implements feedback mechanisms that monitor the locality characteristics of backup data sets. By continuously analyzing data patterns and detecting poor locality conditions, the system can identify when multiplexing has created problematic data distributions and adjust subsequent processing accordingly
Solution Approach 2:
The system changes processing parameters based on detected data characteristics. When poor locality is identified, the system modifies its deduplication approach to account for the disrupted data patterns, adjusting parameters such as processing windows, comparison strategies, and resource allocation to maintain efficiency
3Adaptability or versatility
If data is distributed differently across backup sets due to varying disk loads, then backup flexibility increases, but data locality and deduplication performance worsen
Solution Approach 1:
The system performs preliminary detection and analysis of data distribution patterns across backup sets. By identifying poor locality conditions before deduplication, the system can prepare appropriate handling strategies for the varied data distributions caused by different disk loads and multiplexing patterns
Solution Approach 2:
The deduplication system dynamically adapts to different data distribution patterns encountered in multiplexed backup sets. It adjusts its processing behavior based on the specific locality characteristics of each data set, maintaining performance despite variations in how data is distributed across different backup operations
Data Source
AI summary
Described are computer-based methods and apparatuses, including computer program products, for dynamic deduplication. Data is processed using an algorithm that deduplicates the data based on a first set of parameters. A first moving average of a first deduplication performance metric is calculated for the algorithm over a time period. A second moving average of a second deduplication performance metric is calculated for the algorithm over the time period. It is determined that the first moving average satisfies a first criterion, the second moving average satisfies a second criterion, or both. The algorithm is reconfigured based on a second set of parameters to deduplicate data.