Change Data Event Checkpointing for Exactly-Once Recovery
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Document-oriented databases lack transaction support, making guaranteed delivery of data during recovery difficult in data replication topologies.
Innovation Solution
A system that maintains ordered positions of change data events using a system checkpoint, writes these events to a document-oriented data store, and discards and de-duplicates records to ensure exactly once delivery by leveraging MongoDB's transaction capabilities.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Productivity
If data replication is implemented in a document-oriented database, then data delivery capability is improved, but data delivery reliability deteriorates due to lack of transaction support
Solution Approach 1:
The patent introduces an intermediary mechanism (checkpoint system with ordered positions) between the data source and document-oriented database to track and guarantee delivery. The system maintains a checkpoint collection that records the ordered position of delivered events, acting as a mediator to ensure exactly-once delivery semantics even though the underlying document database lacks native transaction support.
2Reliability
If transaction support is added to document-oriented databases, then data integrity is improved, but device complexity worsens
Solution Approach 1:
The patent segments the transaction guarantee functionality into a separate checkpoint management system rather than requiring full transaction support within the document database itself. The checkpoint collection and delivery tracking logic are separated from the main data storage operations, allowing data integrity to be improved without adding complex transaction mechanisms to the document database core.
3Manufacturing precision
If duplicate records are handled through de-duplication, then data accuracy is improved, but processing time worsens
Solution Approach 1:
The patent performs preliminary action by maintaining a checkpoint collection that records delivered event positions before actual data processing occurs. When recovering or checking for duplicates, the system queries this pre-maintained checkpoint information rather than performing expensive de-duplication operations on the actual data, significantly reducing processing time while maintaining data accuracy.
Data Source
AI summary
A non-transitory computer readable storage medium has instructions executed by a processor to receive change data events from a data source. Ordered positions of the change data events are maintained using a system checkpoint. The change data events are written to a document-oriented data store including a change data event collection and a change data event target checkpoint collection. There is an identification when an ordered position of change data events at the system checkpoint is lower than an ordered position of change data events in the change data event target checkpoint collection. Data source records following a position of change data events at the system checkpoint are collected. Records up to the position of change data events in the change data event target checkpoint collection are discarded and de-duplicated for exactly once delivery.