Multi-node Data De-duplication via Concurrent Fingerprint Analysis
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing multi-node data de-duplication systems face performance bottlenecks due to the need for simultaneous similarity analysis across multiple nodes, which limits concurrent processing and increases data retrieval and locking times.
Innovation Solution
A method where file fingerprint information is sent to multiple meta-data servers (MDSs) to query local fingerprint sets, allowing selection of the most similar group and enabling concurrent similarity analysis, reducing data retrieval and locking times by only querying the responsible MDS's fingerprint sets.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Measurement precision
If fingerprint matching is performed across all groups to ensure querying accuracy, then measurement precision is improved, but device complexity and processing time increase
Solution Approach 1:
The system segments the fingerprint matching process into two stages: first, a quick filtering stage that divides groups into candidate groups and non-candidate groups based on group-level fingerprints; second, a precise matching stage that only queries block fingerprints within selected candidate groups. This segmentation reduces the scope of precise matching while maintaining accuracy.
Solution Approach 2:
The system performs preliminary action by first querying group fingerprints to identify candidate groups before performing the actual block fingerprint matching. This preliminary filtering step eliminates non-candidate groups from further processing, reducing the overall complexity and time required for accurate querying.
2Productivity
If all nodes perform concurrent similarity analysis, then productivity is improved, but resource occupation and locking time increase
Solution Approach 1:
The system segments the concurrency control by allowing multiple nodes to perform similarity analysis concurrently on different candidate groups simultaneously. Each node locks only the specific candidate groups it is processing, rather than requiring global locks, thereby enabling parallel processing without excessive locking time.
Solution Approach 2:
The system applies partial action by having each node perform similarity analysis only on selected candidate groups rather than all groups. This partial processing approach allows multiple nodes to work concurrently on different subsets of data, improving overall productivity while reducing individual node locking time and resource occupation.
3Productivity
If group-based de-duplication is implemented, then productivity is improved by reducing query scope, but manufacturing precision decreases due to limited de-duplication rate
Solution Approach 1:
The system performs preliminary action by first querying group fingerprints to identify candidate groups that are likely to contain duplicate blocks. This preliminary filtering maintains high productivity by limiting the search scope, while the subsequent block-level fingerprint matching within candidate groups ensures high de-duplication accuracy is not compromised.
Solution Approach 2:
The system uses feedback from group-level similarity analysis to guide the block-level fingerprint matching process. Groups with high similarity scores are identified as candidate groups and receive further detailed inspection, while low-similarity groups are excluded. This feedback mechanism ensures that the limited de-duplication rate is applied only where most needed, maintaining both productivity and precision.
Data Source
AI summary
A similarity analysis method, an apparatus, and a system where the method includes acquiring file fingerprint information of a file to be analyzed, sending an analysis request that carries the file fingerprint information to at least two MDSs, selecting at least one group according to an analysis result returned by each MDS, where the analysis result includes a group number and a similarity of at least one group that has the highest similarity with the file fingerprint information and is found by the MDS, and the MDS locally queries a duplicate data block in the selected group. Hence, each MDS needs to query only a file fingerprint information set of a group that the MDS itself is responsible for, which reduces the amount of data retrieval and waiting time of reading, writing, and locking a database file.