Cloud Metadata Synchronization via Multi-Threaded Queue Segmentation
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
In cloud computing environments, synchronizing large amounts of metadata between database instances is time-consuming, often taking hours, which can impact application availability and complicate disaster recovery, especially when secondary instances are activated due to primary database failures or service disruptions.
Innovation Solution
A multi-threaded, asynchronous metadata replication process that spawns multiple discrete connections to the target database instance, synchronizing metadata in concurrent processing threads, allowing for faster synchronization times by allocating tasks based on a threshold value and utilizing separate queues for sequential and asynchronous processing.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Loss of time
If metadata is synchronized sequentially from the primary database instance to the secondary database instance, then the synchronization process is simple to implement, but the synchronization time becomes excessively long (taking hours)
Solution Approach 1:
The patent divides the metadata synchronization process into multiple independent segments or tasks, each handling a specific portion of the metadata. These segmented tasks are then distributed across multiple worker threads that execute concurrently, transforming a single long-running sequential operation into multiple shorter parallel operations, thereby dramatically reducing total synchronization time.
Solution Approach 2:
The patent introduces a new dimension of parallelism by utilizing multi-threading to execute synchronization tasks concurrently rather than sequentially. By spawning multiple worker threads that can run simultaneously, the system transitions from one-dimensional sequential processing to multi-dimensional parallel processing, achieving significant performance acceleration.
2Productivity
If all synchronization tasks are allocated to sequential processing, then resource management is simple, but processing throughput is limited
Solution Approach 1:
The patent segments the synchronization task queue into multiple sub-queues, each dedicated to a specific worker thread. This segmentation allows tasks to be distributed and processed in parallel across multiple threads, increasing throughput while maintaining manageable complexity through organized task distribution.
Solution Approach 2:
The patent implements dynamic task allocation where the system can adaptively assign synchronization tasks to available worker threads based on current system state and workload conditions. This dynamic approach optimizes resource utilization and throughput while maintaining system responsiveness.
Data Source
AI summary
Methods and apparatuses are described for synchronization of metadata between databases in a cloud computing environment. A computing device identifies jobs that synchronize metadata from a first database instance to a second database instance, each job comprising synchronization tasks for a different portion of the metadata. The computing device determines a count of the synchronization tasks for each job. The computing device allocates each job to a first queue where the count of the synchronization tasks for the job is below a threshold value, and allocates the synchronization tasks for each job to a second queue where the count of the synchronization tasks for the job is at or above the threshold value. The computing device executes the jobs in the first queue and executes the synchronization tasks in the second queue to synchronize the corresponding metadata to the second database instance.