Heterogeneous Storage Replication via OST API Instruction Recording
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Data replication between heterogeneous storage systems is inefficient due to Input/Output (I/O) and network traffic issues, as well as backend storage requirements, caused by the use of disparate storage devices and protocols.
Innovation Solution
A method that records instructions during a backup operation, including include and write instructions, to facilitate data replication between heterogeneous storage systems, using a common format enabled by Open Storage Technology (OST) Application Programming Interface (API), allowing for efficient data transfer and storage across different storage devices.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Adaptability or versatility
If data replication is performed between heterogeneous storage systems using traditional methods, then data backup and replication functionality is achieved, but I/O operations and network traffic increase significantly
Solution Approach 1:
The system performs preliminary actions by recording offset ranges and data classification information during the backup operation itself. This pre-processing allows the replication operation to later retrieve only the necessary new data without re-reading entire backup images, thereby reducing I/O operations and network traffic while maintaining compatibility across heterogeneous storage systems
Solution Approach 2:
The invention extracts only the essential information needed for replication by recording offset ranges that identify new data locations and storing classification information that distinguishes new from existing data. This extraction approach allows the system to replicate only necessary data portions between heterogeneous storage systems, minimizing I/O operations and network bandwidth consumption
2Reliability
If complete backup images are transmitted during replication operations, then data integrity is maintained, but network data transfer volume increases
Solution Approach 1:
The backup image is segmented into regions of new data and existing data using offset ranges. During replication, only the segments containing new data (identified by recorded offset ranges) are transmitted to the target storage system, while maintaining data integrity through the use of a common data format facilitated by the OST API
Solution Approach 2:
The system performs preliminary analysis during the backup operation to identify and record offset ranges of new data. This pre-identification allows the replication process to transmit only the necessary data segments rather than complete backup images, reducing network data transfer volume while preserving data integrity through structured transmission
3Reliability
If all backup data is stored during replication operations, then complete data availability is ensured, but backend storage requirements increase
Solution Approach 1:
The system extracts and stores only the classification information and offset ranges that identify new data locations, rather than storing complete backup images. This extraction approach allows the target storage system to retrieve only necessary data portions, ensuring data availability while reducing backend storage requirements
Solution Approach 2:
The source storage system performs preliminary processing during backup to identify new data and record its location information. This pre-processing eliminates the need to store and transfer redundant existing data, thereby reducing backend storage requirements while maintaining complete data availability through efficient data retrieval mechanisms
Data Source
AI summary
Disclosed herein are systems, methods, and processes to perform replication between heterogeneous storage systems. In one embodiment, a request to perform a replication operation is sent by a target server to a source server, where the target server and the source server use different protocols to store data. A plurality of instructions, which are associated with a replication stream, are received from the source server by the target server, where the plurality of instructions comprise an include instruction to include existing data and a write instruction to write new data. A replication stream, which is associated with a backup stream stored on the source server, is also received from the source server, where the replication stream and the backup stream share a common format. The target server synthesizes a new replicated backup image, where the synthesizing comprises performing the include instruction and the write instruction on the replication stream.