Global Deduplication File System with Distributed Metadata Snapshots
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Current deduplication techniques are ineffective when data is distributed across multiple locations, leading to storage inefficiencies and difficulties in ensuring only one user can write to the data while ensuring all users access the latest version, as they struggle to maintain data consistency and performance across different locations.
Innovation Solution
Implementing a global deduplication file system using a deduplication dictionary included in metadata snapshots, which updates and reflects changes across interconnected systems, allowing for incremental data transmission and management of read/write locks to ensure data consistency and performance across multiple locations.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Speed
If data is distributed across multiple locations, then access performance and data availability are improved, but storage efficiency and deduplication effectiveness deteriorate
Solution Approach 1:
The system segments the deduplication dictionary into distributed components stored at each location, allowing local systems to maintain deduplication awareness without requiring centralized control. Each location has its own copy of the distributed deduplication dictionary, enabling parallel operation and improved access performance while maintaining storage efficiency through global deduplication.
Solution Approach 2:
The patent introduces a distributed deduplication dictionary as an intermediary mechanism that mediates between data distribution and deduplication requirements. This intermediary structure enables systems to access data from multiple locations efficiently while simultaneously tracking which data copies are redundant, thus resolving the contradiction between access performance and storage efficiency.
2Quantity of substance
If centralized data storage is used, then deduplication effectiveness is improved, but access performance and system scalability deteriorate
Solution Approach 1:
Instead of a single centralized deduplication dictionary, the system segments it into multiple distributed copies located at different sites. Each location maintains its own segment, allowing local systems to access data quickly without waiting for centralized processing, while the distributed segments collectively provide global deduplication effectiveness.
Solution Approach 2:
The patent changes the parameter of data location from centralized to distributed, transforming the deduplication dictionary from a single point of control to a distributed system. This parameter change enables simultaneous achievement of good deduplication effectiveness (through global coordination) and good access performance (through local availability).
3Reliability
If real-time data synchronization across multiple locations is implemented, then data consistency is improved, but system complexity and bandwidth consumption deteriorate
Solution Approach 1:
The system performs preliminary actions by pre-establishing the distributed deduplication dictionary structure and pre-coordinating the synchronization protocol. This preliminary setup enables automatic and transparent data consistency maintenance without requiring complex real-time intervention, reducing operational complexity while ensuring reliability.
Solution Approach 2:
The patent implements feedback mechanisms where each location periodically updates its local deduplication dictionary segment with changes from other locations. This feedback loop automatically maintains data consistency across the distributed system without requiring complex centralized coordination, balancing reliability with manageable complexity.
4Reliability
If full data transmission to all locations is performed, then data availability is improved, but bandwidth consumption and storage requirements deteriorate
Solution Approach 1:
The system extracts only the necessary information (data blocks that need to be replicated) from the full dataset and selectively transmits them to appropriate locations. The distributed deduplication dictionary identifies which data should be replicated and where, extracting only the essential data for availability while avoiding redundant transmission of data that already exists at a location.
Solution Approach 2:
Each location maintains its own copy of the distributed deduplication dictionary, giving it local quality and autonomy in determining what data needs to be retrieved or replicated. This local awareness enables efficient bandwidth usage by avoiding transmission of data that is already locally available, while still ensuring overall data availability across the distributed system.
Data Source
AI summary
The disclosed embodiments describe interactions and synchronization techniques for global deduplication file systems. In one embodiment of the invention, a method and system implements a global deduplication file system between a plurality of interconnected systems located in different locations globally by making use of the deduplication dictionary included in metadata being periodically snapshot. In yet another embodiment of the invention, a method implements a global deduplication file system between a plurality of interconnected systems located in different locations globally and provides appropriate read/write locks.