Delayed Database Clusters for Data Anomaly Failover
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing database systems fail to provide users with access to historic data during disaster recovery scenarios, leading to prolonged downtime when data corruption occurs.
Innovation Solution
Implementing a delayed database cluster system that includes a primary database for real-time data storage and a delayed database cluster for historic data, with incremental backups and anomaly detection to switch to the delayed cluster during data anomalies, ensuring continuous access to historic data.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If a primary database cluster is used for real-time data storage, then data availability and accessibility are improved, but system reliability deteriorates when data corruption occurs
Solution Approach 1:
The system is divided into two independent database clusters: a primary cluster for real-time data storage and a delayed cluster for historic data. This segmentation allows the system to isolate failures to one cluster while maintaining operational capacity through the other cluster, resolving the contradiction between reliability and downtime.
Solution Approach 2:
A delayed database cluster is created as a copy of the primary cluster, populated with historic data through incremental backups. This copy provides a fallback mechanism that maintains system functionality during primary cluster failures, reducing downtime while preserving reliability.
2Reliability
If incremental backups are performed frequently, then data recovery capability is improved, but storage requirements and system complexity increase
Solution Approach 1:
Incremental backups are performed continuously in advance and stored in a delayed database cluster. This preliminary action ensures that recovery data is ready before failures occur, improving data recovery capability while managing complexity through automated background operations.
Solution Approach 2:
A data management platform acts as an intermediary between the primary database cluster and the delayed cluster, handling backup operations, data synchronization, and failover coordination. This intermediary layer manages system complexity by centralizing control logic and abstracting operational details.
3Productivity
If real-time data is prioritized in the primary cluster, then operational performance is improved, but availability of historic data during failures deteriorates
Solution Approach 1:
The database system is segmented into a primary cluster optimized for real-time operations and a delayed cluster optimized for historic data storage. This segmentation ensures that real-time performance is maintained in the primary cluster while historic data remains available in the delayed cluster during failures.
Solution Approach 2:
Historic data is preliminarily loaded into the delayed cluster through continuous incremental backups. This ensures that historic data is pre-positioned and immediately available during failures, preventing loss of information while maintaining real-time operational performance.
Data Source
AI summary
In some aspects, the techniques described herein relate to a method including: receiving, at a data management platform, incoming data; writing the incoming data to a primary database cluster; executing an incremental backup operation, wherein the incremental backup operation generates an incremental backup of the primary database cluster; storing the incremental backup of the primary database cluster in a storage location; after a predetermined amount of time, executing an incremental restore operation on a delayed database cluster, wherein the incremental restore operation restores the incremental backup of the primary database cluster to the delayed database cluster; determining a data anomaly in the incoming data; and executing a failover command, wherein the failover command redirects database queries from the primary database cluster to the delayed database cluster.