ETL Data Privacy via Automated Masking at Exit Points
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Information integration systems, such as ETL systems, lack effective tools for implementing data privacy regulations, exposing sensitive data to unauthorized access during information integration processing.
Innovation Solution
A method that receives information about the data flow structure of an information integration job, determines sensitive data fields, and applies masking operators before non-trusted target entities to protect sensitive information, ensuring data privacy by disabling or modifying the job execution and using metadata and user authentication to classify trust levels.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Adaptability or versatility
If data is transmitted in unencrypted form through the data flow structure to enable modification and transformation by operators, then data processing flexibility is improved, but data security deteriorates as sensitive data is exposed to unauthorized access
Solution Approach 1:
The system performs preliminary analysis during the compilation phase to identify data exit points where sensitive data leaves the secured data flow structure. Masking operators are automatically inserted at these identified exit points before data is transmitted to target entities, preventing security risks while maintaining unencrypted transmission within the processing pipeline
Solution Approach 2:
Masking operators are introduced as intermediary components between the data flow structure and target entities. These operators selectively mask sensitive data fields only at exit points where data leaves the secured structure, acting as a mediator that protects data during transmission while allowing full processing flexibility within the internal data flow
2Object-affected harmful factors
If masking operators are applied throughout the entire data flow structure to protect sensitive data, then data security is improved, but processing efficiency deteriorates due to unnecessary masking operations
Solution Approach 1:
Instead of applying masking uniformly throughout the entire data flow structure, the system identifies specific local positions (data exit points) where masking is actually needed. The masking operation is applied locally only at these identified exit points where sensitive data leaves the secured structure, avoiding unnecessary masking operations elsewhere in the data flow
Solution Approach 2:
The system performs preliminary analysis during compilation to identify exactly where data exit points are located and which fields contain sensitive information. This preliminary identification allows masking operators to be precisely positioned only where needed, eliminating wasteful processing overhead from unnecessary masking operations
3Reliability
If the system automatically inserts masking operators to protect sensitive data at non-trusted target entities, then data privacy compliance is improved, but system complexity increases due to additional processing steps
Solution Approach 1:
The system performs self-service by automatically analyzing the data flow structure, identifying sensitive data fields, determining data exit points, and inserting appropriate masking operators without requiring manual intervention. The compilation process itself carries out the privacy compliance checks and modifications, eliminating the need for separate complex privacy management systems
Data Source
AI summary
In an approach for providing data privacy in information integration systems, a method performed during compilation of an information integration job receives information regarding a data flow structure of the job to be executed, said data flow structure comprising at least one source system, one or more target entities, and at least one operator for modifying output data provided by the source system. The method determines data exit points at which output data are provided to the target entities and determines at least one non-trusted target entity. The method determines, for each non-trusted target entity, if at least one data field included in the output data provided to the non-trusted target entity is classified as sensitive information, and, if so, modifies the information integration job by including a masking operator directly before a data exit point associated with the non-trusted target entity in order to mask said sensitive information.