File Map Replication via Sequence Node Tracking
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing data deduplication systems face inefficiencies in replication and overwrite processes, leading to unnecessary storage space consumption and incomplete snapshot versions during file replication, due to the transfer of entire file maps and unnecessary reference count updates.
Innovation Solution
Implementing a file map with a sequence file number and replication sequence number, where only updated nodes are replicated, and maintaining a read-only copy of the file map during replication, organized in a tree data structure like a B+ tree, to track changes and reduce unnecessary data transfer.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If the entire file map is transferred during replication, then data consistency is ensured, but storage space consumption increases and transfer efficiency decreases
Solution Approach 1:
The file map is divided into multiple pages, and only the changed pages are identified and transferred during replication. This segmentation allows the system to transfer only the necessary portions of the file map rather than the entire structure, reducing storage space consumption and transfer overhead while maintaining data consistency for the changed portions.
Solution Approach 2:
The patent extracts and identifies only the changed pages within the file map using change tracking mechanisms. By separating the changed pages from the unchanged pages, the system can selectively replicate only the necessary data, avoiding the transfer of redundant unchanged portions and thereby reducing storage space consumption.
2Reliability
If the entire file map is transferred during replication, then data consistency is ensured, but transfer time and network bandwidth consumption increase
Solution Approach 1:
The file map is segmented into multiple pages with change tracking enabled. During replication, only the changed pages are identified and transferred, significantly reducing the total transfer time and network bandwidth consumption compared to transferring the entire file map, while still ensuring data consistency for the modified portions.
Solution Approach 2:
The patent implements partial replication by transferring only the changed pages rather than the entire file map. This partial action approach reduces transfer time and network resource consumption while maintaining sufficient data consistency for the replicated portions, avoiding the excessive action of transferring all pages including unchanged ones.
3Measurement precision
If sequence numbers are updated for all nodes in the file map, then change tracking is accurate, but system complexity and processing overhead increase
Solution Approach 1:
Sequence numbers are updated locally only for the specific nodes that have changed rather than for all nodes in the file map. This local quality approach maintains accurate change tracking for modified portions while avoiding the unnecessary complexity and processing overhead of updating all nodes, thereby improving the efficiency of the change tracking mechanism.
Solution Approach 2:
The file map is segmented into multiple nodes, each with its own sequence number for change tracking. This segmentation allows the system to update sequence numbers independently for only the changed nodes rather than globally for all nodes, reducing system complexity and processing overhead while maintaining accurate change tracking precision.
4Reliability
If reference counts are updated for all duplicate chunks during backup, then deduplication accuracy is maintained, but processing time and computational resources increase
Solution Approach 1:
The patent extracts and identifies only the changed pages during backup operations. By separating changed pages from unchanged pages, the system updates reference counts only for the affected duplicate chunks associated with changed pages, rather than updating all duplicate chunks. This maintains deduplication accuracy for modified data while significantly improving processing efficiency by avoiding redundant updates.
Solution Approach 2:
The backup process implements partial updating by modifying reference counts only for duplicate chunks associated with changed pages rather than updating all duplicate chunks in the file map. This partial action approach maintains sufficient deduplication accuracy for the replicated portions while reducing processing time and computational resource consumption.
Data Source
AI summary
In general, certain embodiments of the present disclosure provide techniques or mechanisms for creating efficient and consistent replication of file overwrites. According to various embodiments, a file map with a sequence file number is created for a data file on a source. The file map includes one or more nodes that each include a sequence node number. When an entry in the file map is changed, the sequence file number is incremented, and the corresponding node is updated by, among other things, setting the sequence node number to equal the current sequence file number. All other nodes in the file map that point to the updated node are similarly updated. When replication occurs, only the updated nodes in the file map are replicated and any unreplicated non-updated nodes are maintained.