Distributed Journal Logging for High-Rate Continuous Data Protection
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Storage systems face bottlenecks when providing continuous data protection (CDP) for high change rate objects, failing to meet specified Recovery Point Objective (RPO) and Recovery Time Objective (RTO) due to insufficient data logging capabilities.
Innovation Solution
A method for dynamically selecting data logs based on real-time metrics such as ingest queue length and CPU utilization to adaptively store data changes, allowing parallel logging across multiple nodes to prevent bottlenecks and ensure timely data ingestion.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Productivity
If data changes are logged in a single data log at a single node, then the system structure is simple, but the storage system becomes a bottleneck and cannot satisfy RPO for high change rate objects
Solution Approach 1:
The patent divides the single data log into multiple data logs distributed across multiple nodes. Each data log stores a portion of the data changes, allowing parallel writing operations. This segmentation increases the overall data logging rate by eliminating the single-node bottleneck while maintaining manageable complexity through distributed architecture.
Solution Approach 2:
The patent transitions from a single-dimensional logging structure (one data log at one node) to a multi-dimensional distributed structure (multiple data logs across multiple nodes). This dimensional expansion allows simultaneous data writing to multiple locations, dramatically increasing throughput for high change rate objects while distributing the system's complexity across multiple independent components.
2Productivity
If data is written to a single node, then the system is easy to manage, but the node becomes overloaded and cannot keep up with high change rates
Solution Approach 1:
The patent segments the data ingestion workload by distributing data changes across multiple nodes instead of concentrating all writes on a single node. Each node handles a subset of the data changes, preventing any single node from becoming overloaded and maintaining high ingestion rates across the distributed system.
Solution Approach 2:
The patent creates a system where multiple nodes can perform the same data logging function simultaneously. Each node is capable of receiving, processing, and storing data changes independently, providing multi-functionality that increases overall ingestion capacity while distributing management responsibilities across the cluster.
3Productivity
If multiple data logs are used across multiple nodes, then the data logging capacity increases, but selecting the appropriate data log becomes more complex
Solution Approach 1:
The patent implements a feedback mechanism where the system continuously monitors the performance and status of multiple nodes, then uses this information to dynamically select the most appropriate data log for storing new data changes. This feedback-driven selection process manages the complexity of multiple data logs by making selection decisions based on real-time system state, ensuring optimal data protection capability.
Solution Approach 2:
The patent introduces dynamic selection of data logs based on current system conditions rather than using a static assignment. The system can adaptively choose which data log to use based on node availability, load conditions, and performance metrics, making the data log selection process flexible and responsive to changing system states while maintaining high data protection capability.
Data Source
Figure 1
Figure 2
Figure 3
AI summary
A set of data changes to a storage associated with a source system is received. for recording the received set of changes, one or more data logs among a plurality of data logs stored in different nodes of a storage system is dynamically selected based at least in part on a dynamic analysis of metrics of the different nodes of the storage system. The data changes are logged in the one or more selected data logs. A reference to a portion of the one or more selected data logs associated with storing the data changes is recorded in a locator register log.