Multi-streaming Synthetic Replication Preserving Synthesis Data
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Conventional data storage systems fail to leverage virtual synthetic replication optimization when replicating virtual synthetic files, as slicing these files into smaller parts loses synthesis information, preventing concurrent data transfers and reducing replication efficiency.
Innovation Solution
Implementing a range replication mechanism that allows clients to specify and replicate only a range of a file, preserving synthesis information and enabling concurrent data transfers without slicing the file into smaller parts, thereby utilizing virtual synthetic replication benefits.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Productivity
If a large file is sliced into multiple smaller files for parallel replication, then replication speed is improved, but synthesis information is lost and virtual synthetic replication optimization cannot be utilized
Solution Approach 1:
The patent divides the large file replication task into multiple parallel streams without physically slicing the file. Each stream replicates a portion of the file data while the system maintains virtual synthetic replication metadata that tracks the original file structure and synthesis information. This allows parallel processing benefits while preserving the ability to reconstruct and optimize based on virtual synthetic relationships.
Solution Approach 2:
The patent introduces a metadata layer that acts as an intermediary between the parallel replication streams and the original file structure. This metadata layer preserves synthesis information and virtual synthetic replication relationships, enabling the system to maintain optimization capabilities while performing parallel replication operations on divided data streams.
2Loss of information
If a single connection is used for file replication, then synthesis information is preserved, but replication efficiency is limited by protocol constraints
Solution Approach 1:
The patent transitions from a single-dimensional sequential replication approach to a multi-dimensional parallel stream approach. By establishing multiple independent replication streams that operate concurrently while maintaining virtual synthetic replication metadata, the system achieves both improved throughput and preserved synthesis information through the added dimension of parallelism.
Solution Approach 2:
The patent creates a multi-functional replication system where multiple parallel streams serve the dual purpose of improving replication throughput while simultaneously preserving virtual synthetic replication optimization capabilities. Each stream is designed to maintain awareness of the original file structure, enabling the system to achieve both efficiency and information preservation.
3Productivity
If virtual synthetic files are sliced into smaller files for replication, then concurrent transfers are enabled, but the VSR optimization cannot be applied
Solution Approach 1:
The patent implements a dynamic replication system where the virtual synthetic replication metadata is maintained throughout the parallel stream replication process. This dynamic metadata structure allows the system to adaptively apply VSR optimization during replication by tracking which portions of the virtual synthetic file have been replicated through which streams, enabling optimization decisions to be made dynamically during the concurrent transfer process.
Data Source
AI summary
According to one embodiment, a request is received to replicate a plurality of ranges of a source file from a source storage system to a target storage system, where the request specifies a first range of the source file and a second range of the source file. A first replication stream and a second replication stream of a replication session are established with the target storage system. First data of the first range and the second data of the second range from the source file are directly read from the source file, without creating separate files for storing the first data and the second data. The first data and second data are transmitted to the target storage system via the first stream and the second stream respectively in parallel.