Progressive Sampling Deduplication Indexing
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Computer systems face inefficiencies in storing large amounts of data due to the need for a large index to store fingerprints of all data partitions, which can exceed the capacity of RAM or disk storage, leading to performance issues with disk access speeds.
Innovation Solution
Implementing a sampling rate to determine which data segment fingerprints are stored in an index memory, allowing only a subset of data blocks to be indexed, and using a cache to prefetch fingerprint and pointer pairs for faster access and to prevent duplicate data storage.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If all data partition fingerprints are stored in the index to enable complete deduplication, then deduplication completeness is improved, but index size exceeds RAM capacity and disk access performance deteriorates
Solution Approach 1:
The patent applies partial action by storing only a subset of fingerprints in the index rather than all fingerprints. The sampling rate parameter (e.g., 1 out of every N blocks) controls the proportion of fingerprints stored, allowing the system to achieve sufficient deduplication effectiveness while keeping the index size within RAM capacity limits.
Solution Approach 2:
The patent changes the parameter of sampling rate to balance between deduplication effectiveness and index size. By adjusting the sampling rate (e.g., changing from 1/N to 1/(N-1)), the system can optimize the trade-off between storing enough fingerprints for effective deduplication and keeping the index small enough for RAM storage.
2Quantity of substance
If the index is stored on disk to accommodate large data volumes, then index capacity is improved, but access speed decreases due to slower disk read rates
Solution Approach 1:
The patent creates a copy of the fingerprint index in RAM by sampling and storing a subset of fingerprints in memory. This in-memory copy enables fast access speeds while the full index remains on disk for capacity requirements, resolving the contradiction between speed and capacity.
3Device complexity
If a sampling rate is applied to reduce index size, then memory requirements and access speed are improved, but deduplication effectiveness may be reduced
Solution Approach 1:
The patent implements feedback mechanisms to monitor and adjust the sampling rate based on actual deduplication performance. The system can detect when the sampling rate is too high (reducing effectiveness) or too low (increasing index size) and dynamically adjust the sampling rate to optimize the balance between index size and deduplication effectiveness.
Data Source
AI summary
A system and method for efficiently reducing a number of duplicate blocks of stored data. A file server both removes duplicate data and prevents duplicate data from being stored in the shared storage. A sampling rate may be used to determine which fingerprints, or hash values, are stored in an index. The sampling rate may be modified in response to changes in characteristics of the system, such as a change in the shared storage size, a change in a utilization of the shared storage, a change in the size of the storage unit, and reaching a threshold corresponding to utilization of the index. Also, a small cache may be maintained for holding fingerprint and pointer pair values prefetched from the shared storage. Each prefetched pair may be associated with data corresponding to a previous hit in the index. The association may be related to spatial locality, temporal locality, or otherwise.