Clone-aware File Replication for Bandwidth Reduction
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing file replication technologies, such as rsync and dd, inefficiently transfer data by making full copies of files even when only a portion is changed and fail to detect shared content, leading to redundant data transfer and storage waste.
Innovation Solution
Implementing hierarchical sharing of data blocks between clone files in a standby filesystem, where only modified data is transferred, and using a clone-aware replication process to identify and synchronize shared data blocks, ensuring efficient synchronization and preserving content sharing between replica files.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If traditional file replication utilities (rsync, scp, tftp) are used to replicate files individually, then file synchronization between filesystems is achieved, but full copies of files are transferred even when only a portion is changed, causing waste of transfer time and storage space
Solution Approach 1:
The patent segments files into data blocks and tracks changes at the block level rather than transferring entire files. The replication system identifies only modified blocks and transfers those selectively, reducing transfer time and storage space while maintaining synchronization reliability.
Solution Approach 2:
The patent changes the fundamental parameter of replication from file-level to block-level. By operating at the block level and tracking modification times, the system can identify what actually needs to be transferred, transforming the replication process from naive full copies to efficient incremental transfers.
2Reliability
If traditional file replication utilities are used, then file synchronization is achieved, but shared content is repeatedly transferred between filesystems for each file containing the shared content, causing redundant data transfer
Solution Approach 1:
The patent merges the replication of shared content into a single operation. When a data block is shared across multiple files in the source filesystem, the system identifies this sharing relationship and transfers the block only once to the destination, then references it for all files that need it, eliminating redundant transfers.
Solution Approach 2:
The patent makes data blocks universal by allowing a single block to serve multiple files simultaneously. The replication system tracks which blocks are shared and ensures they are replicated only once, making the replication process aware of and respecting the shared nature of content across the filesystem.
3Reliability
If traditional file replication utilities are used, then individual file replication is achieved, but content sharing between replica files in the receiving filesystem is lost, causing each transferred file to be treated as independent
Solution Approach 1:
The patent performs preliminary action by establishing the mapping relationships between source files and their shared blocks before replication begins. The system pre-processes the filesystem to identify sharing relationships and uses this information during replication to maintain the same sharing structure in the destination filesystem.
Solution Approach 2:
The patent introduces an intermediary layer of block-level metadata that tracks sharing relationships. This intermediary structure allows the system to preserve content sharing by recording which blocks are shared and by which files, then reconstructing these relationships in the destination filesystem during replication.
4Productivity
If data block-based replication (dd) is used, then replication efficiency is improved, but awareness of files and synchronization history is lost, making incremental replication difficult
Solution Approach 1:
The patent implements feedback mechanisms by tracking modification times and synchronization history at the block level. The system records which blocks have been transferred and when, then uses this feedback information to determine what needs to be transferred in subsequent replication operations, enabling efficient incremental replication.
Solution Approach 2:
The patent makes the replication system dynamic by continuously updating tracking information about file modifications and synchronization status. Rather than static full copies, the system adapts its behavior based on real-time changes and historical data, dynamically determining what needs to be transferred and when.
Data Source
AI summary
In an embodiment, a computer stores source files and source clone files in a source filesystem. The source clone files are shallow copies of the source files, which initially share the same data blocks as the source files. A source file or a source clone file may eventually be modified, where some of the shared blocks are replaced by modified blocks. In a first phase, all data blocks of all source files that are not clones are replicated to a standby filesystem. A second phase compares each clone file on the source filesystem with its base file to detect a set of differed blocks. The second phase copies the differed blocks into the standby filesystem and, in the standby filesystem, applies the differed blocks to a corresponding clone file. Efficiency of keeping the standby filesystem synchronized with the source filesystem is improved by transferring, from the source filesystem to the standby filesystem, only one copy of only data blocks that are modified.