Non-Relational Database Oplog Backup With Parallel Queues
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Backing up operation logs (oplogs) for non-relational databases is not scalable due to latency issues and potential data loss, especially when associating each oplog with its corresponding collection, which involves undesirable latencies and inefficiencies.
Innovation Solution
Utilizing multiple queues and local disk memory to read oplogs into a multitenant queue in working memory, followed by local writer threads organizing them into collection-specific queues before transferring to remote storage, thereby reducing latency and avoiding data loss.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Loss of time
If oplogs are read directly from disk memory and transferred to remote storage, then the backup process is simple, but latency increases and data loss may occur
Solution Approach 1:
The patent segments the oplog backup process into multiple independent stages: reading oplogs from disk, parsing and validating them, organizing into collection-specific queues, and transferring to remote storage. Each stage is handled by dedicated threads (reader threads, parser threads, writer threads), allowing parallel processing and reducing overall latency while maintaining manageable complexity through modular design
Solution Approach 2:
The patent performs preliminary actions by reading and parsing oplogs into working memory queues before the actual backup transfer. The parser threads validate and organize oplog data in advance, ensuring it is ready for efficient transfer. This preliminary processing eliminates the need for sequential disk-read-to-remote-transfer operations, significantly reducing latency
2Reliability
If multiple queues and working memory are used to process oplogs, then latency is reduced and data loss is avoided, but device complexity increases
Solution Approach 1:
The patent introduces working memory queues as intermediary structures between the disk-based oplog storage and the remote storage destination. These queues act as buffers that decouple the reading process from the transfer process, ensuring reliable data flow. The multitenant queue and collection-specific queues serve as intermediaries that organize and manage oplog data systematically, preventing data loss through structured buffering
Solution Approach 2:
The multitenant queue serves multiple functions simultaneously: it acts as a buffer for oplog data, an organization structure for different collections, and a coordination mechanism between reader threads and writer threads. This multi-functional design reduces the need for separate specialized structures, managing complexity while maintaining reliability
3Productivity
If oplogs are transferred to remote storage sequentially, then network connections are not overwhelmed, but backup time increases
Solution Approach 1:
The patent implements dynamic parallel transfer where multiple writer threads can simultaneously transfer oplogs to remote storage based on available network capacity and collection-specific requirements. The system dynamically adjusts the number and priority of active transfer threads, allowing high-speed parallel transfers when network resources are available while preventing overwhelming of network connections through controlled threading
Solution Approach 2:
The patent transitions from sequential single-threaded transfer to multi-dimensional parallel processing by introducing multiple writer threads that operate simultaneously on different collections. This dimensional expansion from 1D (sequential) to ND (parallel) processing significantly increases backup throughput while network connection management handles the complexity through structured concurrency control
Data Source
AI summary
Methods, systems, and devices for data management are described. For example, techniques for scalable backup solutions for non-relational databases are described. Operation logs (oplogs) may capture changes that occur at a non-relational database. A data management system (DMS) may use multiple queues and local disk memory of the host of the non-relational database to streamline the movement of oplogs from the non-relational database to a remote storage environment accessible to the DMS. Oplogs may be parsed into a multitenant queue, moved from the multitenant queue to collection-specific queues, written from the collection-specific queues to local disk memory of the host, and moved from the local disk memory of the host to the remote storage environment. Oplogs from multiple collections may be moved from the collection-specific queues to local disk memory of the host and from the local disk memory to the remote storage environment in parallel, reducing latency.