Storage Server Deduplication Skip Logic
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Storage server resources are wasted due to unnecessary deduplication operations when an external device continuously sends overwrite requests for data blocks that have been previously deduplicated.
Innovation Solution
Implement a method where the storage server determines whether the overwrite quantity of a data block exceeds a threshold within a specific time period, skipping deduplication if it does, and performing it only when the threshold is not exceeded, using a record table to track addresses with high overwrite probabilities and a lookup table for mapping fingerprints.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Loss of energy
If deduplication operation is performed on all received data blocks, then data storage efficiency is improved, but computing resources are wasted when addresses have high overwrite frequency
Solution Approach 1:
The system performs preliminary analysis of address overwrite frequency before executing deduplication operations. By pre-calculating and recording the overwrite quantity for each address within a time period, the system can identify addresses with high overwrite frequency in advance, avoiding unnecessary deduplication processing on these addresses and thus saving computing resources.
Solution Approach 2:
The system applies different deduplication processing strategies to different addresses based on their local characteristics (overwrite frequency). Addresses with high overwrite frequency are excluded from deduplication operations, while addresses with low overwrite frequency continue to receive full deduplication processing. This localized differentiation optimizes resource allocation and improves overall system efficiency.
2Productivity
If overwrite quantity tracking is implemented for all addresses, then resource optimization is improved, but system complexity increases
Solution Approach 1:
The system introduces a threshold parameter (first threshold) for overwrite quantity to simplify decision-making. Instead of complex analysis of each address's characteristics, the system simply compares the overwrite quantity against this threshold. When the overwrite quantity exceeds the threshold, deduplication is skipped; otherwise, normal deduplication processing is performed. This parameter-based approach reduces system complexity while maintaining resource optimization benefits.
3Loss of energy
If deduplication is skipped for high overwrite frequency addresses, then computing resources are conserved, but data deduplication effectiveness decreases
Solution Approach 1:
The system applies deduplication processing partially - not to all addresses, but only to those that meet specific criteria (overwrite quantity within threshold). This partial action approach focuses computational resources on addresses where deduplication will be most effective, while accepting that some addresses (high overwrite frequency) will not receive deduplication processing. The threshold mechanism ensures that the partial action is targeted and effective.
Data Source
AI summary
A data deduplication method includes receiving an overwrite request sent by an external device, where the overwrite request carries a data block and a first address into which the data block is to be stored, determining whether an overwrite quantity of the first address exceeds a first threshold within a time period [t1, t2], where both t1 and t2 are time points, and t2 is later than t1, and when the overwrite quantity of the first address exceeds the first threshold within the time period [t1, t2], skipping performing a deduplication operation on the data block or when the overwrite quantity of the first address does not exceed the first threshold within the time period [t1, t2], performing a deduplication operation on the data block.