Fingerprint Tracking Structure for Faster Deduplication Queries
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing deduplication storage systems face inefficiencies in processing tracking queries due to the impracticality of exhaustive searches across all container indexes, consuming excessive processing time, system bandwidth, and memory space.
Innovation Solution
Implementing a fingerprint tracking structure as a node tree that stores fingerprints and tracking information, allowing for rapid and efficient processing of tracking queries by traversing a predefined and relatively small number of nodes, rather than exhaustively searching all container indexes.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Measurement precision
If exhaustive search across all container indexes is performed to process tracking queries, then complete tracking information can be obtained, but processing time and resource consumption increase excessively
Solution Approach 1:
The patent segments the monolithic container indexes into multiple sharded fingerprint indexes distributed across different storage nodes. Each shard contains a subset of the total fingerprint data, allowing tracking queries to be processed in parallel across multiple nodes rather than requiring a single exhaustive search through all data, thereby reducing processing time while maintaining complete tracking coverage
Solution Approach 2:
The patent pre-computes and stores fingerprint information in a distributed fingerprint index structure before tracking queries are issued. By organizing fingerprints into sharded indexes with predetermined distribution patterns during data ingestion, the system eliminates the need for real-time exhaustive searches, enabling rapid query processing while preserving complete tracking capability
2Measurement precision
If exhaustive search across all container indexes is performed, then all tracking information can be retrieved, but system bandwidth and memory space are consumed excessively
Solution Approach 1:
The patent divides the fingerprint index into multiple sharded segments distributed across different storage nodes. Each shard maintains a portion of the fingerprint data with metadata indicating its content range. This segmentation allows the system to query only relevant shards based on the tracking target, dramatically reducing bandwidth consumption compared to retrieving and searching all container indexes while ensuring complete tracking information is obtained through coordinated shard queries
Solution Approach 2:
The patent implements partial action by querying only the necessary subset of sharded fingerprint indexes required to retrieve complete tracking information, rather than exhaustively processing all container indexes. The system determines which shards contain relevant fingerprints based on data range metadata, processing only those portions needed to answer the tracking query, thus reducing bandwidth and memory usage while maintaining information completeness
3Reliability
If container indexes are used to store all fingerprints, then complete deduplication metadata is maintained, but memory space requirements become impractical
Solution Approach 1:
The patent segments the fingerprint storage into multiple sharded indexes distributed across different storage nodes, with each shard containing a subset of total fingerprints. This segmentation reduces the memory footprint at any single node while collectively maintaining complete deduplication metadata across the distributed system. The sharded structure enables scalable storage by distributing the memory burden across multiple nodes rather than requiring one node to hold all fingerprints
Solution Approach 2:
The patent transitions from a single-dimension container index structure to a multi-dimensional sharded fingerprint index architecture. By introducing the dimension of distribution across multiple storage nodes and organizing fingerprints in a hierarchical shard structure, the system achieves scalable memory usage that grows linearly with data size rather than requiring exponential memory resources, while preserving complete deduplication metadata through coordinated access across shards
Data Source
AI summary
Example implementations relate to deduplication operations in a storage system. An example includes receiving data units to be stored in a persistent storage of a deduplication storage system; generating a fingerprint for each received data unit; identifying a set of fingerprints to be stored in container indexes, the container indexes including metadata indicating storage locations of data units stored in the persistent storage of the deduplication storage system; storing the identified set of fingerprints in at least one container index; and loading fingerprint entries for the identified set of fingerprints in a root node of a fingerprint tracking structure, where the fingerprint tracking structure is to identify each data unit stored in the persistent storage of the deduplication storage system.