Chunk Signature Lookup Using Anchor Blocks for Fast Deduplication
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Data redundancy elimination systems face a tradeoff between large data storage for indexing many data chunks and chunk signatures, which slows down network throughput due to slow lookup operations, especially when smaller chunk sizes are used for higher compression ratios.
Innovation Solution
Implementing a tiered lookup mechanism that uses data sampling and spatial locality to limit the search space for chunk signatures, where anchor chunks with lower occurrence probabilities are selected to map to signature blocks, facilitating fast lookup operations by loading likely matching signatures into a fast-access memory space like RAM.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If large data storage is used to index a large number of data chunks and chunk signatures, then redundancy detection capability is improved, but network throughput deteriorates due to slow lookup operations
Solution Approach 1:
The patent divides the large storage system into two segments: a fast-access memory (RAM) for frequently accessed chunk signatures and a slow-access storage (disk) for the complete signature database. This segmentation allows the system to maintain comprehensive redundancy detection capability while enabling fast lookups for common patterns, thus resolving the throughput bottleneck caused by scanning entire large storage systems.
Solution Approach 2:
The system pre-loads a subset of chunk signatures into fast-access memory before actual data transmission occurs. By preparing this lookup table in advance based on spatial locality analysis and data sampling, the system ensures that when data arrives, the lookup operation can proceed rapidly without blocking network throughput, while still maintaining the ability to detect redundancies against the complete signature database.
2Loss of substance
If smaller chunk sizes are used for higher compression ratios, then compression efficiency is improved, but the number of chunks to be indexed increases, slowing down lookup operations
Solution Approach 1:
Instead of loading all chunk signatures into fast-access memory, the system loads only a partial subset that is most likely to be needed based on data sampling and spatial locality analysis. This partial action approach maintains high compression ratios by detecting redundancies for the loaded subset while avoiding the time penalty of loading and searching the entire signature database, thus resolving the contradiction between fine-grained chunking and lookup speed.
Solution Approach 2:
The system dynamically adjusts the chunk size parameter based on the data characteristics being processed. By analyzing spatial locality and data patterns, the system can select optimal chunk sizes that balance compression efficiency with lookup performance, rather than using fixed small chunk sizes that always prioritize compression over speed.
Data Source
AI summary
A data redundancy elimination system. In particular implementations, a method includes accessing a data block; dividing the data block into a plurality of data chunks; computing chunk signatures for the plurality of data chunks; identifying a plurality of anchor chunks in the plurality of data chunks; accessing a second memory to identify one or more matching chunk signatures; reading one or more signature blocks corresponding to the matching chunk signatures from a first memory into the signature block search space of the second memory; and matching one or more computed chunk signatures to chunk signatures of the one or more signature blocks in the signature block search space.