Data Ingestion Failover Replication for Cloud Transfer Recovery
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing data transfer techniques in cloud-based database systems are susceptible to system failures during data upload, leading to potential data loss and inefficiencies in disaster recovery, particularly when source and target tables are in different cloud providers.
Innovation Solution
Implementing auto-ingestion techniques with pre-configured primary and secondary deployments, where only the primary deployment processes data transfer requests, and in case of a failover, a secondary deployment seamlessly takes over based on replication information from the old primary deployment, ensuring continuous data transfer.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If data transfer is performed using traditional upload techniques in cloud-based database systems, then data can be transferred from source to target tables, but the system is susceptible to failures during data upload leading to data loss and requiring manual intervention for disaster recovery
Solution Approach 1:
The patent implements pre-configured primary and secondary deployments before data transfer operations begin. The secondary deployment is prepared in advance with the capability to take over if the primary deployment fails, eliminating the need for manual disaster recovery intervention and ensuring continuous data transfer capability.
Solution Approach 2:
The patent creates a secondary deployment that replicates the primary deployment's data transfer capability. This copy serves as a backup system that can assume the primary's responsibilities upon failure, thereby improving reliability without requiring complete system redesign.
2Reliability
If a secondary deployment is pre-configured for disaster recovery, then seamless failover can occur improving reliability, but the device complexity and resource requirements increase
Solution Approach 1:
The secondary deployment is designed with multi-functionality, serving both as a standby for disaster recovery and as a potential primary deployment when needed. This universal design reduces overall system complexity by using identical configurations for both primary and secondary roles, rather than requiring specialized failover infrastructure.
Solution Approach 2:
The system implements automatic failover detection and execution, where the secondary deployment autonomously takes over when primary deployment fails. This self-service mechanism eliminates the need for complex manual intervention systems and reduces the operational complexity of managing disaster recovery.
3Productivity
If data transfer operations are monitored and managed manually, then control over the process can be maintained, but latency increases and productivity decreases
Solution Approach 1:
The patent implements automatic monitoring of data transfer operations with real-time feedback mechanisms. When the primary deployment experiences failure, the system automatically detects the failure condition and triggers failover to the secondary deployment, eliminating delays associated with manual monitoring and intervention.
Solution Approach 2:
The system ensures continuous data transfer operation by maintaining the secondary deployment in a ready state that can immediately assume the primary's responsibilities. This continuity eliminates interruptions and reduces overall transfer latency, as the data transfer process never fully stops during failover events.
Data Source
AI summary
Described herein are techniques for improving disaster recovery, in particular disaster recovery pertaining to data transfer requests. The data transfer request can be received by each of multiple deployments; however, only a primary deployment can process the request. The data transferred by the primary deployment may be replicated in the secondary deployments. In response to a failover event, one of the secondary deployments can be designated as the new primary development and continue the data transfer based on the data transfer request and the replication information received from the old primary deployment prior to the failover.