Asynchronous Volume Replication Logging via Overlap Detection
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing data replication systems face challenges in minimizing resource overhead, such as CPU, memory, and I/O bandwidth, which can impact application performance due to the need to store both metadata and data in replication logs, leading to disk contention and inefficient logging mechanisms.
Innovation Solution
A system and method for efficient logging that only stores data in the replication log when a write request overlaps with earlier write requests, using a data change map to track changes and reduce memory requirements, allowing for periodic replication of changes to secondary storage.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If both metadata and data are stored in the replication log for every write request, then data consistency and reliability are improved, but resource overhead (CPU, memory, I/O bandwidth) increases and application performance deteriorates
Solution Approach 1:
The patent extracts only the necessary portion of data (specifically, only data that actually changed and needs replication) from the full write request, storing it separately in the replication log rather than storing all metadata and data uniformly. This selective extraction reduces the volume of data written to the log while maintaining the reliability needed for consistent replication.
Solution Approach 2:
The patent applies different storage strategies to different portions of data based on their replication needs. Changed data blocks are stored in the replication log with metadata, while unchanged blocks are not written to the log at all. This local differentiation optimizes resource usage by applying quality control only where necessary rather than uniformly across all data.
2Reliability
If data is stored in the replication log for every write request, then data protection is improved, but disk I/O load doubles and creates disk contention
Solution Approach 1:
The patent applies partial action by writing to the replication log only when necessary (i.e., when data blocks have changed and need to be replicated). Instead of performing the full write operation (metadata + data) for every request, the system performs only the partial write of changed data blocks, reducing disk I/O load while maintaining adequate data protection for replication purposes.
3Productivity
If a separate set of disks is allocated for logging to eliminate disk contention, then write throughput is improved, but device complexity and administrative overhead increase
Solution Approach 1:
The patent merges the logging function with the existing storage infrastructure by writing changed data blocks to the same disk devices that store the volume data. Instead of requiring separate dedicated logging disks, the system combines data storage and replication logging onto shared storage resources, reducing device complexity while managing disk contention through selective writing of only changed blocks.
4Productivity
If memory buffering is used to reduce disk reads, then logging efficiency is improved, but memory requirements increase and buffer overflow occurs with large pending changes
Solution Approach 1:
The patent employs a data change map that tracks which data blocks have changed and need replication. This map acts as a lightweight, temporary tracking mechanism that requires minimal memory compared to buffering all pending changes. The map is periodically cleared and rebuilt as replication catches up, allowing the system to maintain high logging efficiency without requiring large amounts of persistent memory or risking buffer overflow with extensive pending changes.
Data Source
AI summary
A system and method for logging for asynchronously replicating volume groups. A write request to write data to a location in a volume may be received. Metadata associated with the write request may be stored. It may be determined if the write request possibly overlaps with one or more earlier write requests to the volume that have not yet been replicated to a secondary storage. The data may be stored in a replication log only if the write request possibly overlaps with one or more earlier write requests to the volume. The data may not be stored in the replication log if the write request does not overlap with one or more earlier write requests to the volume. The data may be written to the location in the volume. Changes to the volume may periodically be replicated to the secondary storage using the replication log.