Blockprint-Based Data Replication for Bandwidth-Constrained Networks
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing remote replication strategies face challenges due to network throughput limitations, especially for small to medium-sized enterprises, as the data production rate often exceeds available network capacities, and techniques like compression and delta replication are insufficient to manage the data transfer efficiently.
Innovation Solution
The use of cryptographic hash functions to identify unique blocks of data, allowing only non-duplicate blocks to be transmitted across the network by creating blockprints, which are then used to reconstruct files at the destination, along with a blockprint database to optimize storage and reduce network traffic.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Productivity
If remote replication transfers complete data files over the network, then data replication is achieved, but network throughput capacity is exceeded and network performance degrades
Solution Approach 1:
The patent divides data files into discrete blocks of fixed size (e.g., 4KB, 8KB, or 16KB). Each block is independently processed to generate a blockprint, and only blocks that need to be replicated are transmitted over the network. This segmentation allows the system to transfer only necessary data portions rather than complete files, reducing network traffic volume while maintaining replication productivity.
Solution Approach 2:
The patent extracts and transmits only the essential identification information (blockprints) rather than the complete data blocks. The blockprint serves as a compact representation that identifies unique blocks, allowing the system to determine what needs to be replicated without transferring unnecessary data. This extraction approach significantly reduces the quantity of data transmitted over the network.
2Quantity of substance
If compression techniques are applied to reduce bandwidth, then network capacity utilization improves, but compression processing time increases and throughput is reduced
Solution Approach 1:
Instead of compressing data blocks, the patent creates a copy of the block's identification information (blockprint) that represents the block's unique characteristics. This blockprint copy is much smaller than the original data block and can be transmitted efficiently over the network. The blockprint serves as a fingerprint that allows the receiving end to identify and reconstruct only the necessary data portions, eliminating the need for time-consuming compression while reducing bandwidth consumption.
3Productivity
If delta replication compares previous and current file versions, then data transfer efficiency improves, but the process only works after initial copy and cannot handle new file creation
Solution Approach 1:
The patent performs preliminary action by pre-computing blockprints for all blocks in a file and storing them in a database before the replication process. When replication is needed, the system queries this pre-computed blockprint database to quickly identify which blocks need to be replicated, eliminating the need for time-consuming comparison operations during the actual data transfer. This preliminary preparation enables efficient handling of both updates and new file creations.
Solution Approach 2:
The blockprint database serves multiple functions: it stores identification information for existing blocks, enables quick determination of blocks needing replication, and works for both updating existing files and creating new files. This universal approach allows the same mechanism to handle different replication scenarios (updates, deletions, new files) without requiring separate processing logic, thereby improving adaptability while maintaining efficiency.
4Quantity of substance
If the system transmits only necessary data blocks, then network traffic is reduced, but the system complexity increases due to blockprint generation and database management
Solution Approach 1:
The patent changes the parameter of data representation from raw data blocks to their cryptographic hash values (blockprints). This parameter transformation reduces the size of data that needs to be processed and transmitted, as blockprints are much smaller than the original blocks. The system processes only the blockprint parameters rather than the full data, which simplifies the replication decision-making process while reducing network traffic. The trade-off is acceptable because the blockprint generation is a one-time computation that enables efficient subsequent replication operations.
Data Source
AI summary
A system and method for communicating a data file between networked computers is disclosed. The method includes the operation of requesting a data file consisting of a plurality of file blocks to be transferred from a source computing device to a destination computing device. A hash signature for each of the plurality of file blocks of the data file is sent from the source computing device to the destination computing device. It can be determined which of the file blocks of the data file are already located on the destination computing device based on the hash signature of each file block. The remaining file blocks can be transferred from the source to the destination computing device. The file can be constructed at the destination computing device from the plurality of file blocks.