In-Memory Data Replication via Manifest Synchronization
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Conventional data replication techniques face challenges in achieving fast data access, updating, and recovery simultaneously, with lazy propagation leading to slow failover and eager replication resulting in slow data updating, making them impractical for high-reliability and high-throughput settings.
Innovation Solution
An in-memory data processing system using volatile memory to store and manage data entities, employing a manifest-based resynchronization technique that synchronizes data entities based on version numbers, allowing for fast updating, access, and recovery while efficiently using network resources.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Speed
If lazy propagation techniques are used to synchronize data entities, then fast data updating is achieved, but failover becomes slow because copies must be updated from logs before access can be provided
Solution Approach 1:
The patent segments the data entity into a primary copy and multiple backup copies stored in volatile memory across different devices. Each copy can be independently accessed, allowing the system to provide fast failover by switching to a pre-synchronized backup copy without requiring log replay, while maintaining fast updating through efficient change propagation mechanisms.
Solution Approach 2:
The patent implements preliminary synchronization where backup copies are proactively updated in volatile memory before failover is needed. This preliminary action ensures that backup copies are already current when the primary device becomes unavailable, enabling immediate failover without the delay of reconstructing data from logs during the failover process.
2Reliability
If eager replication techniques are used to synchronize data entities, then fast failover is achieved because all copies are kept synchronized, but data updating becomes slow because copies must be updated before the data entity itself
Solution Approach 1:
The patent inverts the conventional eager replication approach by allowing the primary data entity to be updated first in volatile memory, and then propagating changes to backup copies. This inversion maintains fast updating by not blocking the primary update operation while still achieving synchronized backups through efficient change propagation, resolving the contradiction between update speed and synchronization.
3Reliability
If data is stored in non-volatile memory to ensure data integrity, then reliability is improved, but access and updating become slow due to time-consuming reads and writes
Solution Approach 1:
The patent creates volatile memory copies of data entities stored in non-volatile memory. These copies reside in fast volatile memory on multiple devices, enabling rapid access and updating operations. The system maintains data integrity by ensuring copies are synchronized with the authoritative non-volatile storage, allowing fast operations without sacrificing reliability.
Solution Approach 2:
The patent changes the storage parameter from exclusively non-volatile memory to a hybrid architecture using volatile memory for active data copies. This parameter change enables high-speed access and updating operations in volatile memory while maintaining the reliability guarantees of non-volatile storage through periodic synchronization and consistency protocols.
Data Source
AI summary
A data processing system configured to store a plurality of data entities in volatile memories of multiple different computing devices. The data processing system comprises a first computing device having a first volatile memory configured to store a first data entity; and a second computing device having a second volatile memory configured to store a copy of the first data entity. The first computing device is configured to perform: receiving an indication to update the first data entity; after receiving the indication, updating the first data entity in the first volatile memory, and providing to the second computing device an indication to update the copy of the first data entity; and providing an indication that the first data entity has been updated, after receiving information from the second computing device indicating that the copy of the first data entity has been updated in the second volatile memory.