Merkle Tree Reference Counts for Deduplication Index Management
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Conventional deduplication systems face issues with scalability and performance due to the need for a large and resource-intensive dedupe index, which becomes inaccessible if damaged, and requires significant processing power to search, especially in systems with limited resources.
Innovation Solution
Implementing a data location map separate from the dedupe index, allowing data to be accessed independently of the index, and using reference counts in Merkle trees to efficiently manage and clean up nodes, thereby reducing resource requirements and enabling flexible deduplication algorithm changes.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Quantity of substance
If a large dedupe index is used to store pointers to deduplicated data, then data deduplication capability is improved, but system resource consumption and processing power requirements increase significantly
Solution Approach 1:
The patent divides the traditional single dedupe index into two separate components: a dedupe index storing only data commonality information and a data location map storing location information. This segmentation allows each component to be optimized independently, reducing the processing burden on the dedupe index while maintaining deduplication capability.
Solution Approach 2:
The patent extracts location information from the dedupe index and places it in a separate data location map. This extraction removes the resource-intensive location tracking function from the dedupe index, allowing the index to focus only on deduplication logic and reducing overall system resource consumption.
2Quantity of substance
If a large dedupe index is used to store pointers to deduplicated data, then data deduplication capability is improved, but search time and resources required to find deduplicated data increase
Solution Approach 1:
By segmenting the index into dedupe index and data location map, the patent enables independent optimization of search operations. The data location map can be structured for efficient location queries without burdening the dedupe index with location tracking logic.
Solution Approach 2:
The data location map acts as an intermediary between the dedupe index and the actual data storage locations. This intermediary layer handles location resolution efficiently, reducing the time required to find deduplicated data without affecting the deduplication capability of the index.
3Use of energy by moving object
If the dedupe index is made volatile to save resources, then resource consumption is reduced, but data accessibility is lost if the index is lost or altered
Solution Approach 1:
The patent extracts location information into a separate data location map that can be stored persistently. This allows the dedupe index to be volatile while the location map remains persistent, ensuring data accessibility even when the index is lost or altered.
Solution Approach 2:
The patent applies different quality characteristics to different components: the dedupe index can be volatile with lower durability requirements, while the data location map is made persistent to ensure reliability. This local differentiation of quality attributes optimizes both resource consumption and data accessibility.
4Productivity
If reference counts are implemented in Merkle trees to manage node cleanup, then node management efficiency is improved, but device complexity increases
Solution Approach 1:
The patent implements reference counts in Merkle tree nodes that automatically track usage and enable self-service cleanup. Nodes with zero reference counts are automatically identified and removed, improving node management efficiency without requiring complex external management mechanisms.
Data Source
AI summary
A method, article of manufacture, and apparatus for managing data commonality in a Merkle tree is disclosed. Reference counts are associated with a node in the Merkle tree. Data commonality is detected and the root of the detected data commonality is determined. If a node is the root node of the detected data commonality, the nodes reference count is incremented. When an object is deleted, the root node of the object is determined. The reference count of the node is decremented. If the count reaches zero, the node is removed from the Merkle tree, and its child nodes are decremented.