Non-Relational Database Backup Using Parallel Buffer Pools
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Non-relational databases face scalability issues with per-collection backups, leading to undesirable latencies and inefficiencies in data management systems (DMS) due to the large number of collections and asynchronous replication across hosts.
Innovation Solution
The DMS employs different buffer pools for data extraction and movement, using extractor and mover jobs to iteratively fill and reuse buffers, allowing parallel processing of multiple collections and hosts, and utilizes oplogs for capturing additional changes to ensure synchronization and consistency in backups.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If per-collection backups are used for non-relational databases, then backup completeness is improved, but backup time and latency increase significantly
Solution Approach 1:
The backup system segments the backup process into two independent parallel streams: (1) collection-level backup that backs up individual collections from multiple hosts, and (2) database-level backup that captures the entire database state. This segmentation allows the system to maintain complete backup coverage while reducing overall backup time by executing multiple backup operations simultaneously rather than sequentially through all collections.
Solution Approach 2:
The patent introduces a new dimension to the backup approach by implementing database-level snapshots in addition to traditional collection-level backups. This adds a hierarchical layer (database level) above the collection level, enabling parallel execution of backups at different granularities and thus reducing total backup time while maintaining completeness.
2Reliability
If per-collection backups are implemented across multiple hosts, then data coverage is improved, but system complexity and coordination overhead increase
Solution Approach 1:
The patent merges multiple backup operations (collection backups from different hosts) into a unified database-level snapshot process. By combining these operations at the database level, the system achieves comprehensive data coverage across all hosts while simplifying coordination through a single snapshot mechanism rather than managing complex inter-host collection backup synchronization.
Solution Approach 2:
The database-level snapshot mechanism serves multiple functions simultaneously: it backs up collections across all hosts, captures the global database state, and provides a unified recovery point. This multi-functional approach improves data coverage while reducing system complexity by replacing multiple specialized backup processes with a single universal snapshot capability.
3Reliability
If asynchronous replication is used across hosts, then data consistency is improved, but backup synchronization latency increases
Solution Approach 1:
The system performs preliminary actions by taking database-level snapshots at specific points in time before initiating collection-level backups. These preliminary snapshots establish known good states across all hosts, allowing subsequent backup operations to proceed without waiting for asynchronous replication to complete, thus reducing synchronization latency while maintaining data consistency.
Data Source
AI summary
Methods, systems, and devices for data management are described. For example, techniques for scalable backup solutions for non-relational databases are described. A data management system (DMS) may use different buffer pools for the extraction of data from the non-relational database and the movement (e.g., copying or transferring of data) from the buffers to a remote storage location. A backup process for a collection may involve one or more iterations involving retrieving or selecting a buffer from an empty buffer queue of an agent of the DMS at the non-relational database, filling the buffer with data from the non-relational database to the retrieved buffer, moving the filled buffer to a full buffer queue, moving the data to a remote storage location, and moving the buffer back to the empty buffer queue after moving the data on the buffer to the remote storage location.