Logical Data Replication via Unordered Object Streams
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Conventional data replication systems face limitations, particularly in replicating data at the logical block level, where they require identical disk topology and directory structures between source and destination storage systems, leading to inefficiencies and performance issues due to the need to process directory updates before file updates.
Innovation Solution
The method involves replicating data containers at the logical level in an unordered stream of individual data units, preserving object identifiers to allow for out-of-order processing, and creating snapshots to identify and transfer only modified directory entries, thus bypassing the need for maintaining identical disk geometries and directory structures.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If data replication is performed at the logical block level requiring identical disk topology and directory structures, then data consistency is maintained, but system flexibility and adaptability are reduced
Solution Approach 1:
The patent segments the replication process into independent operations for directory entries and file data. Directory entries are replicated first to establish the structure, then file data is replicated independently. This segmentation allows the destination system to receive and process data without requiring identical disk topology or directory structures, resolving the contradiction between maintaining consistency and providing system flexibility.
Solution Approach 2:
The patent performs preliminary replication of directory entries before replicating file data. By establishing the directory structure first at the destination system, subsequent file data replication can proceed independently without requiring the destination to have pre-existing matching structures. This preliminary action enables flexibility in disk topology while maintaining data consistency.
2Stability of the object's composition
If directory updates are processed before file updates in conventional replication, then directory structure integrity is maintained, but replication speed and productivity are reduced
Solution Approach 1:
The patent performs preliminary replication of directory entries to establish the directory structure at the destination system before replicating file data. This preliminary action ensures directory structure integrity is maintained while enabling subsequent file data replication to proceed independently and in parallel, thereby improving replication speed without compromising structural integrity.
Solution Approach 2:
The patent introduces dynamic processing where directory entries and file data are replicated in an optimized sequence rather than rigidly. The system can dynamically adjust the replication flow, processing directory updates first when necessary for structure integrity, but allowing file updates to proceed independently afterward, thus improving overall productivity while maintaining stability.
3Reliability
If all data is transferred in ordered fashion to maintain directory structure, then data integrity is ensured, but bandwidth efficiency and processing requirements increase
Solution Approach 1:
The patent segments data replication into two distinct phases: directory entry replication and file data replication. This segmentation allows directory structures to be established first with smaller data transfers, then file data to be replicated in an optimized manner. The segmentation improves bandwidth efficiency by avoiding the transfer of redundant directory information with each file update while maintaining data integrity through the structured approach.
Solution Approach 2:
The patent extracts and separates directory entry replication from file data replication. By taking out the directory structure replication as a distinct preliminary step, the system can then replicate file data independently without requiring ordered transfer of all data. This extraction improves bandwidth efficiency by eliminating redundant transfers while maintaining data integrity through the separated processing approach.
Data Source
AI summary
A method and apparatus for replicating a data container of a source storage server at the logical level in an unordered stream of individual data units are described. In certain embodiments, the replication operations can be performed without regard to the differences in geometry of physical persistent storage of the source and destination storage servers and without regard to the differences in data container format of the source and destination storage servers. An initial replication operation of the source data container is performed by transferring the data objects within the source data container to a replica data container in the destination storage server as an unordered stream of individual data units while preserving data object identifiers of the source data objects. Afterwards, incremental replication operations can be performed to capture modifications to the source data container over time after the initial replication operation.