Asynchronous Data Journaling in Hybrid Cloud
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Conventional hybrid cloud systems using synchronous distributed replicated block devices (DRBD) for database journaling are costly, slow, and limited in scalability and bandwidth due to the need for constant active instances, slower virtual network links, and restricted replication to a fixed number of clouds, leading to potential data loss in case of failures.
Innovation Solution
An asynchronous data journaling model utilizing a cloud queueing service where virtual machines write journal records to a journaling device, which are then converted to messages by a hypervisor and published by the cloud queueing service, allowing asynchronous replication across secondary clouds using a host OS's storage network, reducing latency and enabling high scalability and bandwidth.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If synchronous distributed replicated block devices (DRBD) are used for database journaling, then data consistency is maintained, but system latency increases and scalability is limited
Solution Approach 1:
The system segments the journaling process into two independent parts: (1) local journal writing performed synchronously by the virtual machine to ensure data consistency, and (2) remote replication performed asynchronously by a separate replication service. This segmentation allows the primary VM to complete transactions without waiting for remote cloud acknowledgments, reducing latency while maintaining data integrity through the local journal.
Solution Approach 2:
A replication service acts as an intermediary between the primary virtual machine and secondary cloud storage. The replication service receives journal data from the primary VM and handles all remote replication operations, including converting block data to messages and publishing them to cloud queues. This intermediary isolates the primary VM from remote network delays, eliminating their impact on transaction speed.
2Reliability
If synchronous replication to multiple clouds is implemented, then data redundancy is improved, but device complexity and cost increase
Solution Approach 1:
The replication service operates autonomously to manage all aspects of multi-cloud replication. It automatically converts block data to messages, publishes them to appropriate cloud queues, and handles retransmission logic. The system uses existing cloud infrastructure services (queueing services, storage) to perform replication tasks, eliminating the need for complex custom replication protocols and reducing overall system complexity.
Solution Approach 2:
The replication service is designed as a universal component that can replicate journal data to any number of secondary clouds using standard message queue interfaces. It handles multiple cloud providers and storage systems through a unified approach, converting block data to messages that can be published to any cloud queueing service, thereby supporting data redundancy across multiple clouds without proportionally increasing complexity.
3Ease of operation
If virtual network links are used for cloud-to-cloud communication, then setup is simplified, but bandwidth and speed are reduced
Solution Approach 1:
The system replaces direct virtual network link communication with a message queue-based architecture. Instead of establishing persistent high-speed network connections between clouds, the replication service publishes journal data as messages to cloud queueing services. Secondary clouds subscribe to these queues and retrieve messages asynchronously. This substitution uses standardized messaging protocols that are easier to configure and manage while achieving high effective bandwidth through optimized message transmission.
4Ease of manufacture
If a fixed number of cloud replicas are maintained, then resource allocation is simplified, but adaptability and scalability are limited
Solution Approach 1:
The replication service dynamically adapts to changes in the number of required replicas without requiring system reconfiguration. It monitors subscription requests from secondary clouds and automatically adjusts replication operations accordingly. The service can scale from replicating to one cloud to multiple clouds seamlessly, with each secondary cloud independently subscribing to the journal data queue. This dynamic approach allows flexible resource allocation while maintaining simplicity through standardized message-based communication.
Data Source
AI summary
A system and method for scheduling of hyper-threaded CPUs using memory monitoring includes a memory with an operating system memory and a physical processor in communication with the memory. The physical processor includes a first hyper-thread and a second hyper-thread. A monitor instruction to monitor for updates to a designated memory location is executed in the first hyper-thread. The system further includes an operating system to execute on the physical processor and a system call configured to record in the operating system memory that the first hyper-thread is in a memory wait state. The system call is further configured to execute a memory wait instruction in the first hyper-thread. A task is executed in the second hyper-thread while the first hyper-thread is in the memory wait state.