Log-Structured Journal for Distributed Database Data Protection
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
In high-performance distributed database services, it is challenging to efficiently manage workload associated with secondary tasks like index management, backup, and restore without impacting the performance of service nodes responsible for fulfilling client requests.
Innovation Solution
The implementation of a log-structured journal service that stores records of committed state changes of database objects, allowing for continuous data protection and global secondary indexes, thereby separating the management of these tasks from the primary database service nodes.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If secondary tasks like index management, backup and restore are added to service nodes responsible for fulfilling client requests, then data protection and index management capabilities are improved, but system performance and availability deteriorate
Solution Approach 1:
The system segments service nodes into primary nodes that handle client requests and secondary tasks (index management, backup, restore) that are handled by separate secondary service nodes. This segmentation allows primary nodes to maintain high performance for client operations while secondary nodes handle data protection and maintenance tasks independently.
Solution Approach 2:
Secondary tasks (index management, backup, restore) are extracted from primary service nodes and delegated to dedicated secondary service nodes. This extraction removes the workload burden from primary nodes, ensuring they can fulfill client requests without performance degradation while secondary nodes specialize in data protection functions.
2Reliability
If more service nodes are allocated to handle secondary tasks, then data protection and index management are improved, but device complexity and resource allocation become more challenging
Solution Approach 1:
Secondary service nodes are designed with multi-functionality to handle various secondary tasks including index management, backup operations, and restore operations. This universal design reduces the need for separate specialized nodes for each function, simplifying overall system architecture while maintaining comprehensive data protection capabilities.
Solution Approach 2:
The system introduces an intermediary coordination mechanism between primary service nodes and secondary service nodes, allowing efficient task delegation and communication. This intermediary layer manages the complexity of coordinating multiple secondary tasks across distributed nodes without requiring complex point-to-point management between all components.
3Device complexity
If workload for secondary tasks is concentrated on fewer nodes, then device complexity is reduced, but those nodes become bottlenecks affecting overall system availability
Solution Approach 1:
The system segments secondary tasks across multiple dedicated secondary service nodes rather than concentrating them on few nodes. Each secondary node handles specific subsets of secondary tasks, distributing the workload to prevent bottlenecks while maintaining simplified management through clear role separation between primary and secondary nodes.
Solution Approach 2:
The system implements dynamic workload distribution among secondary service nodes, allowing tasks to be allocated and balanced based on current system conditions and node capacity. This dynamic approach prevents any single node from becoming a bottleneck while maintaining operational simplicity through automated load management.
Data Source
AI summary
In response to determining that continuous data protection is to be enabled for a particular table of a database service, a service component verifies that automated transmission of change records of the table to a log-structured journal has been configured. A given change record comprises a before-image and an after-image associated with a committed database write, and is assigned a unique sequence number. In response to a determination to restore the table as of a specified point in time, a restore record set is identified from the journal with respect to a selected snapshot of the table. The restore record set includes change records which are not represented in the snapshot and are to be represented in the restored table. A restore result table is created using the selected snapshot and the restore record set.