Metadata Log Replication for Asynchronous Storage Systems
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing data storage systems face high processing costs and overhead in performing remote replication, particularly due to the expensive snap diff technique and cache misses during data access.
Innovation Solution
The proposed solution utilizes an existing metadata (MD) log to track logged entries that have not been replicated, introducing a new MD log tuple type (LA to PLB) for direct mapping of logical addresses to physical storage locations, thereby avoiding the need for a separate persistent replication log.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If the snap diff technique is used for remote replication, then data consistency is ensured, but processing cost and overhead increase significantly
Solution Approach 1:
The patent extracts only the essential metadata (logical address to physical block mapping) from the full snapshot data, storing only the differential changes in the MD log rather than complete data sets. This selective extraction maintains data consistency verification while dramatically reducing processing overhead and memory requirements.
Solution Approach 2:
Instead of creating full snapshots for replication, the system creates lightweight copies of only the metadata log entries that track physical block locations. These MD log copies enable verification of data consistency without the computational expense of traditional snapshot-based replication methods.
2Speed
If data is accessed from cache during replication, then replication speed improves, but cache misses cause performance degradation
Solution Approach 1:
The system performs preliminary actions by pre-processing write operations and immediately logging the logical address to physical block mapping in the MD log before data is written to storage. This preliminary metadata logging ensures that when replication occurs, the physical block locations are already known and can be directly accessed without cache misses, improving both speed and efficiency.
3Reliability
If a separate persistent replication log is created, then replication reliability improves, but system complexity and storage overhead increase
Solution Approach 1:
The MD log serves multiple functions simultaneously: it acts as the replication log, the snapshot differential log, and the metadata tracking structure. By making the MD log multi-functional, the system eliminates the need for separate persistent replication logs while maintaining replication reliability and reducing overall system complexity.
Solution Approach 2:
The patent merges the replication log functionality with the existing MD log structure. Instead of maintaining separate logs for replication tracking and metadata management, the system combines these functions into a single unified MD log that stores both the mapping information and replication state, thereby reducing system complexity while preserving reliability.
Data Source
AI summary
Data replication techniques can include receiving, at a source system, a write directed to a source logical device configured for asynchronous remote replication to a destination system; performing processing that flushes a transaction log entry for the write; and performing replication processing that uses a replication queue including a replication queue entry corresponding to the write that stores the first content to a logical address. The processing can create a metadata (MD) log entry in a MD log for the write responsive to determining that the write is directed to the source logical device configured for asynchronous remote replication and that the first content has not been replicated. Responsive to the first content not being in cache, the first content can be retrieved using the reference to a storage location storing the first content. The reference can be obtained from the MID log entry or the replication queue entry.