Sticky Node Routing for Data Chunk Deduplication
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
In multi-node data deduplication systems, existing methods face inefficiencies in routing data streams, particularly with interleaved data, leading to poor deduplication performance and increased storage space usage due to the bid-based approach, which fails to effectively route similar data segments to the same back-end node, resulting in redundant storage.
Innovation Solution
The implementation of a 'sticky routing' algorithm that designates a 'sticky node' for each data stream, ensuring large contiguous pieces of new data are stored on a single back-end node, and dynamically changes the sticky node based on data accumulation, using bid and sticky thresholds to optimize deduplication efficiency.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Ease of operation
If bid-based routing is used to distribute data across multiple back-end nodes, then load balancing is improved, but deduplication performance deteriorates because similar data segments are not routed to the same node
Solution Approach 1:
The system performs preliminary actions by designating a sticky node for each data stream before data is actually routed. This pre-established routing relationship ensures that subsequent data segments are sent to the same node, enabling effective deduplication while maintaining load balance across nodes.
Solution Approach 2:
The patent replaces the bid-based mechanical routing system with a sticky routing system. Instead of using bids to determine node selection, the system uses predetermined sticky node assignments that are maintained across multiple data segments, eliminating the need for continuous bid evaluation while improving deduplication efficiency.
2Volume of stationary object
If data is distributed across multiple back-end nodes, then storage capacity is improved, but storage efficiency deteriorates due to redundant storage of duplicate chunks
Solution Approach 1:
The system merges the routing of related data segments to the same back-end node by using sticky node designation. This combining of data streams at the same destination enables the deduplication system to recognize and eliminate redundant chunks, improving storage efficiency while maintaining distributed storage capacity.
3Quantity of substance
If sparse indexing or Bloom filter methods are used to limit memory usage, then memory consumption is reduced, but access time increases when consulting the full chunk index
Solution Approach 1:
The system applies local quality by maintaining different indexing strategies at different nodes. Each back-end node maintains its own chunk index locally, allowing for efficient local access without requiring centralized index consultation. This distributed indexing approach reduces both memory consumption at each node and access time by eliminating centralized bottlenecks.
Data Source
Figure 1a
Figure 1b
Figure 2
AI summary
A method includes receiving information about a plurality of data chunks and determining if one or more of a plurality of back-end nodes already stores more than a threshold amount of the plurality of data chunks where one of the plurality of back-end nodes is designated as a sticky node. The method further includes, responsive to determining that none of the plurality of back-end nodes already stores more than a threshold amount of the plurality of data chunks, deduplicating the plurality of data chunks against the back-end node designated as the sticky node. Finally, the method includes, responsive to an amount of data being processed, designating a different back-end node as the sticky node.