Selective Data De-duplication via Yield Scoring
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing data de-duplication methods often result in inefficient metadata growth and resource consumption by blindly de-duplicating all data, even when substantial storage savings are not achieved, leading to increased costs without sufficient benefits.
Innovation Solution
An intelligent de-duplication approach that scores data based on potential storage savings, selectively de-duplicating only high-yield data items, and using techniques like fingerprinting with SHA1 or Rabin fingerprinting to identify and prioritize data for de-duplication, thereby optimizing metadata management and resource utilization.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Quantity of substance
If all data is de-duplicated, then storage savings are maximized, but metadata size and processing costs increase excessively
Solution Approach 1:
The patent applies local quality by differentiating data items based on their de-duplication yield. High-yield data items (those with multiple copies) receive de-duplication treatment while low-yield items do not. This selective approach optimizes metadata usage by concentrating resources on data items that provide substantial storage savings, thereby avoiding excessive metadata growth from processing all data uniformly.
Solution Approach 2:
The patent changes the parameter of de-duplication applicability from universal to selective based on yield thresholds. By introducing a yield threshold parameter, the system dynamically determines which data items qualify for de-duplication processing. This parameter change enables the system to balance storage savings benefits against metadata overhead by only processing data items that exceed the yield threshold.
2Quantity of substance
If de-duplication is performed on all data, then potential storage savings increase, but system resources are consumed inefficiently
Solution Approach 1:
The patent applies partial action by performing de-duplication only on high-yield data items rather than all data. The system calculates potential storage savings for each data item and selectively processes only those exceeding a predefined yield threshold. This partial approach ensures that system resources are allocated efficiently to data items that provide substantial storage benefits, avoiding waste on low-yield items.
Solution Approach 2:
The patent applies local quality by differentiating data items based on their de-duplication yield. High-yield data items (those with multiple copies and substantial savings potential) receive de-duplication treatment while low-yield items do not. This selective approach optimizes metadata usage by concentrating resources on data items that provide substantial storage savings, thereby avoiding excessive metadata growth from processing all data uniformly.
3Adaptability or versatility
If fingerprinting is performed on all data, then de-duplication coverage increases, but metadata storage requirements expand
Solution Approach 1:
The patent applies local quality by differentiating data items based on their de-duplication yield. High-yield data items (those with multiple copies) receive de-duplication treatment while low-yield items do not. This selective approach optimizes metadata usage by concentrating resources on data items that provide substantial storage savings, thereby avoiding excessive metadata growth from processing all data uniformly.
Solution Approach 2:
The patent changes the parameter of de-duplication applicability from universal to selective based on yield thresholds. By introducing a yield threshold parameter, the system dynamically determines which data items qualify for de-duplication processing. This parameter change enables the system to balance storage savings benefits against metadata overhead by only processing data items that exceed the yield threshold.
Data Source
AI summary
Disclosed is an improved approach for implementing de-duplication, by selecting data such that the de-duplication efficacy of the storage will be increased without arbitrarily increasing metadata size.