Synchronizing Masking Jobs Across Engines
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing data masking technologies face challenges in ensuring consistent masking across different masking engines, leading to inconsistent results when multiple engines are used for backup, disaster recovery, or horizontal scaling, as they may operate with different versions or random number sequences, affecting debugging and analysis.
Innovation Solution
A method for synchronizing masking jobs between masking engines by identifying and exporting dependent objects, generating an export payload with revision information, and importing it into other engines to ensure consistent masking, using an import/export module to manage the synchronization of job objects, algorithms, and connectors.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If multiple masking engines are deployed for backup, disaster recovery, or horizontal scaling, then system reliability and productivity are improved, but masking consistency deteriorates due to different versions or random number sequences
Solution Approach 1:
The masking job is divided into multiple independent objects (job objects, algorithm objects, rule set objects, connector objects, file format objects) that can be independently exported, imported, and synchronized between masking engines. This segmentation allows each engine to maintain its own instances while ensuring consistency through controlled distribution of these objects.
Solution Approach 2:
Revision information is added to track changes in masking objects across different engines. By comparing revision identifiers, the system determines whether objects need to be synchronized, ensuring that parameter changes (such as algorithm modifications or rule updates) are consistently propagated across all masking engines while allowing them to operate independently.
2Productivity
If multiple masking engines operate independently, then processing capacity and availability are improved, but result consistency deteriorates
Solution Approach 1:
Multiple masking engines are designed to perform the same masking functions using the same job objects, algorithm objects, and rule set objects. This universality ensures that each engine can independently process data with identical masking logic, maintaining result consistency while enabling parallel processing and load distribution across multiple engines.
Solution Approach 2:
Masking objects (job objects, algorithm objects, rule set objects, connector objects, file format objects) are copied from one masking engine to another through export and import operations. This copying mechanism ensures that identical versions of masking configurations are available across multiple engines, enabling consistent results while allowing each engine to operate independently with the same masking parameters.
3Manufacturing precision
If masking objects are synchronized between engines, then masking consistency is improved, but system complexity increases due to dependency management
Solution Approach 1:
An export/import mechanism serves as an intermediary between masking engines for synchronizing objects. The first masking engine exports masking objects (job objects, algorithm objects, rule set objects, connector objects, file format objects) with their revision information to a second masking engine, which then imports and compares these objects. This intermediary process manages dependencies systematically without requiring direct complex inter-engine communication or centralized coordination.
Solution Approach 2:
Revision information is used as feedback to determine whether masking objects need to be synchronized between engines. By comparing revision identifiers of exported and local objects, the system automatically determines whether updates are needed, reducing unnecessary synchronization operations and simplifying the management of dependencies across multiple masking engines.
Data Source
AI summary
A data processing system maintaining a plurality of masking engines is configured to synchronize masking jobs between different masking engines, in order to ensure consistent outputs between the different engines for the same job. Upon identifying a masking job object maintained by a first masking engine to be exported, dependencies of the job object are traversed to identify all objects depended on by the job object, which are used to form an export payload. At least object of the payload is a reference object that references other objects maintained by the first masking engine. When importing the export payload onto a second masking engine, the export payload is analyzed to identify revision information of the referenced object. The identified revision information is compared with revision information of a corresponding object maintained by the second masking engine, and the referenced objects is imported based upon a result of the comparison.