Granular Data Replication via B+ Tree Tag Filtering
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Current data replication methods lack the fine-grained control and efficiency needed for modern data systems, particularly in cloud storage systems, as they struggle to balance the granularity of per-file transfers with the performance of bulk filesystem transfers while maintaining consistency and resource utilization.
Innovation Solution
A granular data replication process that utilizes a B+ Tree structure and tag filtering to replicate files at the sub-directory level, allowing for efficient data movement by assigning metadata tags to files and directing them to proper locations, enabling fast and reliable replication with fine-grained control.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Manufacturing precision
If per-file data movement is used, then fine-grained control is achieved, but throughput is slow
Solution Approach 1:
The patent segments the filesystem into subdirectory-level units that can be independently replicated. Instead of treating the entire filesystem as one bulk object or individual files, it creates intermediate replication units at the subdirectory level, allowing selective replication of only changed subdirectories while maintaining bulk transfer efficiency for each unit.
Solution Approach 2:
The patent introduces a new dimension of granularity control by implementing subdirectory-level segmentation between file-level and filesystem-level operations. This intermediate level allows the system to achieve both fine-grained control and bulk transfer performance by operating on subdirectory units rather than individual files or entire filesystems.
2Productivity
If bulk filesystem transfer is used, then throughput is increased, but fine-grained control is lost
Solution Approach 1:
The patent divides the filesystem into manageable subdirectory units that can be independently processed. This segmentation enables the system to perform bulk transfers at the subdirectory level rather than requiring entire filesystem replication, achieving both efficiency and selectivity.
Solution Approach 2:
The patent implements dynamic replication where the granularity of transfer units adapts based on change detection. Subdirectories that have not changed are skipped entirely, while changed subdirectories are replicated in bulk. This dynamic approach optimizes throughput by transferring only necessary data units at appropriate granularities.
3Reliability
If per-file replication is used, then consistency is maintained, but resource utilization is inefficient
Solution Approach 1:
The patent merges multiple file-level operations into subdirectory-level bulk operations. By combining files within a subdirectory into a single replication unit, the system reduces the number of individual transfer operations while maintaining consistency through change detection at the subdirectory level.
Solution Approach 2:
The patent implements partial replication by identifying and replicating only the subdirectories that have changed since the last replication. This partial action approach maintains data consistency by replicating changed units while skipping unchanged ones, significantly reducing resource consumption compared to full filesystem replication.
4Productivity
If subdirectory-level replication is implemented, then efficiency is improved, but system complexity increases
Solution Approach 1:
The patent performs preliminary change detection at the subdirectory level before initiating replication. By pre-identifying which subdirectories have changed using metadata comparison, the system prepares a replication plan that avoids unnecessary transfers, simplifying the actual replication process while maintaining efficiency.
Solution Approach 2:
The patent introduces subdirectory metadata as an intermediary layer between files and the replication system. This metadata tracks changes at the subdirectory level and serves as a mediator to determine which units require replication, reducing the complexity of tracking individual file changes while maintaining fine-grained control.
Data Source
AI summary
Embodiments for granular replication of data with high efficiency. A defined metadata element embodied as a tag is assigned to each file. Tag filtering is used to direct the data to the proper location. Files with different tags can be selected for transfer. Embodiments can be used with a defined backup system file replication process, such as present in the Data Domain File System. By using snapshots, incoming new data is continued to be ingested while the replication is in process and maintaining data consistency at the same time. This is achieved by performing operations on B+ Tree snapshots in conjunction with tag filtering on keys present in the leaf pages of these structures. This method efficiently makes a single pass walk of a B+ Tree in contrast with previous methods that look up files one-by-one via their pathname.