Data Synchronization via Redundancy Detection and Block Clustering
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Current methods for transferring data between cloud datacenters are inefficient, requiring excessive time and bandwidth due to the lack of consideration for data similarities across multiple data sets, especially in scenarios involving virtual machine images, software packages, and archives.
Innovation Solution
A method that identifies similarities between data sets across multiple datacenters, creates clusters based on shared data blocks, and generates an optimal data replication plan to minimize bandwidth usage and transfer time by selecting the most efficient source servers based on geographical proximity, cost, and trust factors.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Productivity
If data is transferred between datacenters using conventional methods, then data replication is achieved, but excessive bandwidth and time are consumed due to lack of redundancy detection
Solution Approach 1:
The system performs preliminary analysis of data sets before transfer by computing fingerprints and identifying redundant data blocks. This preliminary action enables the replication process to skip already-synchronized data, significantly reducing bandwidth consumption and transfer time while maintaining data consistency across datacenters.
Solution Approach 2:
The invention changes the parameter representation of data by computing fingerprints (hash values) of data blocks and comparing these parameters rather than transferring entire data sets. This parameter transformation enables efficient redundancy detection and allows the system to identify and skip duplicate data blocks, improving transfer efficiency and reducing bandwidth usage.
2Loss of time
If data blocks are distributed across multiple servers without redundancy detection, then storage distribution is achieved, but transfer time increases due to inability to leverage existing copies
Solution Approach 1:
The system implements feedback mechanisms by maintaining metadata about data block locations across servers and using this information to guide future transfer operations. The fingerprint comparison results feed back into the replication decision process, enabling the system to adaptively select source and target servers based on current data distribution state, thereby reducing transfer time while managing complexity through structured metadata.
Solution Approach 2:
The invention segments data into discrete blocks and computes fingerprints for each block independently. This segmentation allows the system to analyze and transfer only the necessary portions of data rather than entire data sets, enabling parallel processing of multiple blocks and significantly reducing overall transfer time while distributing storage across multiple servers efficiently.
3Reliability
If all data blocks are transferred to ensure completeness, then data integrity is maintained, but bandwidth usage increases due to redundant data transfer
Solution Approach 1:
The system performs preliminary fingerprint comparison between source and target data blocks before initiating transfer. This preliminary action identifies redundant data blocks that already exist at the target location, allowing the system to skip transferring these blocks while maintaining data completeness. Only unique or updated blocks are transferred, ensuring reliability while minimizing bandwidth usage.
Solution Approach 2:
The invention transforms the data representation by using fingerprints (hash parameters) instead of raw data for comparison purposes. This parameter change enables efficient identification of redundant blocks through cryptographic hash comparison, which is computationally inexpensive compared to actual data transfer. The system maintains data integrity by verifying fingerprint matches while transferring only necessary data blocks.
Data Source
AI summary
Managing data in a cloud computing environment, including data transfers. File level and block level similarities are identified, including for archive and nested archive files, residing on datacenters and regional repositories. A replication plan is generated based on receiving a replication instruction, and further based on similarity clusters by transferring unique data blocks and files from best available sources including regional repositories.