Late-ETL Data Replication for Disaster Recovery
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Conventional disaster recovery systems for large data centers, especially in big data environments, face challenges due to the sheer size of data storage, where terabytes of non-transactional daily changes are common, and do not support efficient point record updates, making recovery from disasters like power outages difficult.
Innovation Solution
A geographically apart master-slave architecture with a late-ETL stage data replication model and Map-Reduce-based replication implementation is used, along with data cluster monitoring and workload balancing, to ensure high availability and disaster recovery in large-scale data warehouses, supporting petabytes or exabytes of data with always-on service.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If conventional commit log based disaster recovery systems are used, then transactional data integrity can be maintained, but recovery becomes infeasible for big data systems with terabytes of non-transactional daily changes
Solution Approach 1:
The system changes the fundamental parameters of data replication by abandoning transactional commit logs and point-update mechanisms for a bulk data replication approach using Map-Reduce frameworks. This allows the system to handle terabytes of non-transactional daily changes by processing data in distributed batches rather than individual transactions, making disaster recovery feasible for big data environments while maintaining data integrity through consistent state replication.
Solution Approach 2:
The patent replaces the mechanical commit log scanning and point-update system with a Map-Reduce based bulk processing mechanism. Instead of linearly scanning logs and applying individual updates, the system uses distributed computing frameworks to replicate entire data sets or large portions of data in parallel, substituting the traditional transactional processing mechanics with a scalable distributed computing approach suitable for petabyte-scale data warehouses.
2Reliability
If data is replicated across geographically separate sites, then disaster recovery capability is improved, but data consistency and replication overhead increase
Solution Approach 1:
The system segments the data replication process into discrete Map-Reduce jobs that can be independently executed and managed. By dividing the large-scale data replication into smaller, manageable tasks that process data in distributed batches, the system reduces the complexity of managing geographically distributed replication while maintaining data consistency through coordinated job execution across multiple data centers.
Solution Approach 2:
The patent introduces an intermediary layer of metadata management and job coordination that mediates between the source and target data centers. This intermediary mechanism tracks replication state, manages job scheduling, and ensures consistency without requiring direct complex point-to-point synchronization, thereby simplifying the overall replication system architecture while enabling geographic distribution for disaster recovery.
3Productivity
If linear log scanning is used for recovery, then update efficiency is improved, but recovery speed becomes insufficient for large data centers with massive data volumes
Solution Approach 1:
The system transitions from one-dimensional linear log scanning to multi-dimensional parallel processing by leveraging the Map-Reduce framework. Instead of processing recovery data sequentially through linear log scans, the system executes multiple replication jobs in parallel across distributed computing nodes, adding a temporal and spatial dimension to the recovery process that dramatically increases recovery speed for petabyte-scale data warehouses.
Solution Approach 2:
The patent implements partial replication strategies where only necessary portions of data are replicated or updated in each job, rather than requiring complete resynchronization. This allows the system to make progress on recovery through multiple partial actions that collectively achieve full data restoration, significantly accelerating the recovery process compared to waiting for complete linear log processing of entire data sets.
Data Source
AI summary
Techniques and mechanisms described herein facilitate the management of data centers. According to various embodiments, a first query dispatcher at a first data center may be disabled. The first data center may include a first data cluster and a first database. The first query dispatcher may be configured to dispatch queries to access data in the first data cluster and in a second data cluster at a second data center. The second data cluster may replicate data stored in the first data cluster. Metadata stored in the first database may be replicated to a second database in the second data center. The metadata may describe a state of the first data cluster. A second query dispatcher may be enabled at the second data center. The second query dispatcher may be configured to dispatch queries to access data in the second data cluster.