Cluster Deduplication via Zone-Based Sketch Routing
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
In cluster deduplication systems, the need to send sampled metadata information to all physical nodes for querying leads to increased interactions and calculation loads, resulting in degraded performance as the number of nodes increases, particularly when the number of physical nodes exceeds 16.
Innovation Solution
The method involves dividing data streams into blocks, obtaining fingerprint sketch values, and identifying corresponding physical nodes using algorithms to reduce the number of queries, thereby minimizing the calculation load and maintaining performance regardless of the number of nodes in the cluster system.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Measurement precision
If sampled metadata information is sent to all physical nodes for querying, then deduplication accuracy is improved, but system performance degrades due to increased interactions and calculation load
Solution Approach 1:
The patent segments the cluster system into multiple zones, each responsible for specific physical nodes. This segmentation allows metadata queries to be directed to specific zones rather than all nodes, reducing the scope of interactions while maintaining effective deduplication search capability.
Solution Approach 2:
The patent introduces zone information as an intermediary layer between data nodes and the querying process. Zone information acts as a mediator that routes metadata queries to appropriate zones, reducing the number of direct interactions required between physical nodes while maintaining query effectiveness.
2Quantity of substance
If the number of physical nodes in the cluster system is increased, then storage capacity is improved, but calculation amount increases leading to performance degradation
Solution Approach 1:
The patent divides the cluster into zones with each zone managing specific nodes. This segmentation ensures that as the total number of nodes increases, the query scope remains manageable by limiting interactions to within zones, thereby maintaining performance while supporting larger storage capacity.
Solution Approach 2:
Zone information serves as an intermediary that enables efficient routing of queries in large-scale cluster systems. It allows the system to scale by adding more nodes to existing zones or creating new zones, maintaining manageable calculation amounts while increasing overall storage capacity.
3Measurement precision
If metadata information is queried from all physical nodes, then duplicate data detection accuracy is improved, but the number of interactions between nodes increases
Solution Approach 1:
The patent segments the node network into zones, reducing the interaction scope from all nodes to only relevant nodes within zones. This maintains effective duplicate detection while significantly reducing the number of interactions required between nodes.
Solution Approach 2:
Zone information acts as an intermediary that filters and routes metadata queries to appropriate zones only, reducing unnecessary interactions between nodes while maintaining comprehensive duplicate data detection capability through the zone-based metadata indexing structure.
Data Source
Figure 1
Figure 2~3
Figure 4
AI summary
In embodiments of the present invention, when a duplicate data query is performed on a received data stream, a first physical node which corresponds to each first sketch value and is in a cluster system is identified according to a first sketch value representing the data stream, and then the first sketch value representing the data stream is sent to the identified physical node for the duplicate data query, and a procedure of the duplicate data query does not change with an increase of the number of nodes in the cluster system; therefore, a calculation amount of each node does not increase with an increase of the number of nodes in the cluster system.