Incremental Masking Engine for Real-Time Database Replication
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Current masking systems operate in batch processing, making it time-consuming and expensive to provide users with reasonably recent versions of masked datasets, as the entire dataset must be re-masked every time it is updated.
Innovation Solution
Implementing a masking engine that receives change logs from a source database, applies a masking algorithm, and stores masked changes in a buffer to apply them incrementally to a masked replica database in real-time, reducing the need for frequent full masking processes.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Productivity
If batch processing is used to mask the entire dataset, then data masking can be completed with simple processing logic, but the time required to provide updated masked datasets increases significantly
Solution Approach 1:
The patent segments the masking process into two distinct components: (1) initial batch masking of the complete dataset, and (2) incremental masking of only the changed portions. This segmentation allows the system to leverage batch processing for comprehensive coverage while using incremental processing for rapid updates, thereby resolving the contradiction between completeness and speed.
Solution Approach 2:
The patent performs preliminary batch masking of the entire dataset before any incremental updates. This preliminary action creates a baseline masked dataset that can be quickly updated later by applying only the changes detected through change data capture, eliminating the need to re-mask the entire dataset for subsequent updates.
2Reliability
If the entire dataset is re-masked every time it is updated, then data security and consistency are maintained, but computational resources and costs increase
Solution Approach 1:
The patent extracts only the changed portions of the dataset using change data capture technology, separating the static (unchanged) portions from the dynamic (changed) portions. This extraction allows the system to apply masking resources only to the necessary changed data while maintaining consistency with the original dataset, thereby reducing computational resource consumption without compromising data security.
Solution Approach 2:
Instead of applying excessive action by re-masking the entire dataset, the patent applies partial action by masking only the changed portions identified through change data capture. This partial masking approach maintains data consistency and security while significantly reducing the computational resources and costs required for the masking operation.
3Speed
If change logs are processed immediately without buffering, then real-time masking is achieved, but conflicts between concurrent transactions may occur
Solution Approach 1:
The patent performs preliminary buffering of change logs in a pre-buffer before applying them to the masked dataset. This preliminary buffering allows the system to collect and validate complete transactions, ensuring that only consistent, conflict-free changes are applied. This approach maintains real-time masking capability while preventing transaction conflicts through proper sequencing and validation.
Solution Approach 2:
The pre-buffer acts as an intermediary between change log generation and application to the masked dataset. This intermediary component buffers incoming change logs, validates transaction completeness, and sequences changes to prevent conflicts, thereby enabling real-time masking while maintaining transaction consistency and reliability.
4Loss of time
If masked change logs are applied incrementally, then update time is reduced, but the complexity of managing buffers and conflict detection increases
Solution Approach 1:
The patent segments the buffer management into two distinct buffers: a pre-buffer for collecting and validating change logs, and a main buffer for storing masked change logs pending application. This segmentation simplifies the complexity of buffer management by clearly defining the purpose and contents of each buffer, making the system easier to manage while enabling efficient incremental masking.
Data Source
AI summary
A system performs masking of data stored in a database. The system receives change logs from a source database and masks the change logs to generate masked change logs. The system stores the masked change logs in a pre-buffer. If the system determines that the set of masked change logs stored in the pre-buffer corresponds to a complete transaction, the system determines whether any conflicts exist between masked change logs stored in the pre-buffer and masked change logs stored in a main buffer. If the system identifies conflicts, the system applies the masked change logs stored in the main buffer to a masked replica database. This causes the system to move masked change logs corresponding to a complete transaction from the pre-buffer to the main buffer.