Storage System Dynamic Sampling for Deduplication Overhead
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
The existing deduplication systems face performance degradation due to the increasing number of fingerprints in the fingerprint management information, leading to high overhead in search and update operations, which can be costly and inefficient.
Innovation Solution
A storage system is configured to divide data into chunks and select characteristic chunk data based on a sampling period, calculating fingerprints only for these chunks to determine duplication, with the sampling period adjusted based on past deduplication processes, thereby reducing the number of fingerprints registered and improving deduplication performance.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Loss of substance
If fingerprints are calculated for all data items to ensure accurate deduplication, then data reduction effect is improved, but the number of fingerprints registered increases leading to performance degradation
Solution Approach 1:
The patent segments the data set into multiple chunks and selectively calculates fingerprints for only certain chunks (e.g., first chunk, last chunk, or chunks at regular intervals) rather than all chunks. This segmentation approach maintains adequate deduplication detection capability while significantly reducing the number of fingerprints that need to be registered and processed, thus resolving the contradiction between data reduction effect and deduplication performance
Solution Approach 2:
The patent applies partial action by calculating fingerprints for only a subset of chunks rather than all chunks. By selecting specific chunks (such as boundary chunks or periodically sampled chunks) for fingerprint calculation, the system achieves sufficient deduplication coverage without the overhead of processing every single chunk, thereby improving deduplication performance while maintaining acceptable data reduction effects
2Measurement precision
If the sampling period is shortened to calculate fingerprints for more chunks, then deduplication accuracy is improved, but the overhead of fingerprint management increases
Solution Approach 1:
The patent introduces a dynamic sampling period that can be adjusted based on deduplication requirements and performance considerations. The sampling period is not fixed but can be modified to balance between deduplication accuracy and management overhead, allowing the system to adapt to different workloads and storage conditions, thus resolving the contradiction between accuracy and time overhead
Data Source
AI summary
This storage system is designed to: divide data into a plurality of chunk data (pieces of data) in a deduplication process; select one or more chunk data from among the plurality of chunk data in accordance with a sampling period which indicates that, on average, one chunk data be selected from among each N chunk data; and calculate a fingerprint, such as a hash value, for each of one or more characteristic chunk data, which are the selected one or more chunk data, and determine whether data including the one or more characteristic chunk data is a duplication. The storage system changes the sampling period on the basis of the results of past deduplication processes.