Filtered Data Replication in Distributed Data Centers
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
In distributed database systems, replicating all data from a primary to a secondary data center leads to network traffic overload and resource inefficiency, as not all data is necessary for redundancy and disaster recovery.
Innovation Solution
Implementing a replication service unit that filters and encrypts only modified, relevant data specific to the primary data center, which can be recreated or is essential for the secondary data center, thereby reducing unnecessary data replication and optimizing resource usage.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If all data is replicated from primary to secondary data center, then data redundancy and disaster recovery capability are improved, but network traffic increases and secondary data center becomes overloaded
Solution Approach 1:
The patent extracts only the necessary subset of data for replication by filtering out data that can be rebuilt locally at the secondary data center. This selective extraction approach maintains disaster recovery capability while significantly reducing network traffic and resource consumption during replication operations.
Solution Approach 2:
Instead of replicating all data (excessive action), the system performs partial replication by identifying and copying only the essential data portions that cannot be regenerated at the secondary location. This partial action approach achieves sufficient redundancy without the overhead of complete data duplication.
2Reliability
If all data is replicated from primary to secondary data center, then data redundancy is improved, but resource efficiency deteriorates
Solution Approach 1:
The system extracts and identifies data that can be rebuilt at the secondary data center, separating it from data that must be replicated. This extraction process improves resource efficiency by avoiding the waste of bandwidth, storage, and processing resources on duplicating reconstructible data, while still maintaining necessary redundancy for critical data.
Solution Approach 2:
The patent changes the parameter of data selection from 'all data' to 'necessary data only' by introducing filtering criteria based on data reconstructibility. This parameter change optimizes resource efficiency by adjusting what gets replicated based on its essentiality and reconstructibility characteristics.
3Loss of energy
If data filtering is implemented to reduce replication, then network traffic is reduced, but system complexity increases
Solution Approach 1:
The system performs preliminary action by pre-tagging and categorizing data at the primary data center according to its reconstructibility and essentiality before replication occurs. This preliminary classification enables efficient filtering during replication without requiring complex real-time analysis, thereby reducing network traffic while keeping the system complexity manageable.
Solution Approach 2:
The patent introduces an intermediary component (data classification service or metadata layer) that manages the complexity of data filtering. This intermediary handles the sophisticated logic of determining which data to replicate, shielding the core replication system from complexity while achieving traffic reduction through intelligent filtering.
Data Source
AI summary
Various embodiments of systems, computer program products, and methods to replicate data in a distributed environment are described herein. In an aspect, the data from a first persistent storage unit associated with a primary data center is received. Further, at least a part of the received data is determined to be replicated by filtering the data based on one or more predetermined conditions. The determined part of the data is transmitted to replicate in a second persistent storage unit associated with a secondary data center to recover the part of the data during failure of the primary data center.