Data Anonymization System Using ML Entity Extraction
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Current technologies face challenges in effectively masking personal and confidential information from data for sharing, particularly due to privacy and security concerns, which hinders data sharing for machine learning and artificial intelligence model development, testing, and cloud processing.
Innovation Solution
A machine learning and rule-based system for identifying, anonymizing, and de-anonymizing sensitive structured and unstructured data, utilizing techniques like entity extraction, replacement strategies, and human-in-the-loop feedback for custom masking, supports multiple languages and maintains data consistency, enabling secure data sharing and model training.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If data masking is applied to remove personal and confidential information, then data privacy and security are improved, but data utility for machine learning and AI model development deteriorates
Solution Approach 1:
The system creates synthetic copies of real data that preserve statistical properties and patterns while removing identifiable personal information. Synthetic data generation techniques produce artificial datasets that maintain the utility needed for machine learning and AI model development without containing actual sensitive information from individuals.
Solution Approach 2:
The system transforms data by changing parameters such as perturbing numerical values, generalizing categorical data, and modifying data structures while preserving the underlying patterns and relationships. These parameter changes allow data to maintain its analytical value for ML/AI purposes while eliminating privacy risks associated with direct identification.
2Reliability
If comprehensive data masking is applied to ensure complete anonymization, then data protection compliance is improved, but data quality and consistency deteriorate
Solution Approach 1:
The system applies different masking techniques to different parts of the dataset based on sensitivity levels and usage requirements. Critical personally identifiable information receives strong anonymization, while other fields maintain higher quality through lighter masking approaches, preserving overall data consistency and utility.
Solution Approach 2:
The masking approach is made dynamic and adaptable rather than static. The system can adjust the level of masking based on the specific context, purpose of data usage, and risk assessment, allowing data quality to be optimized for different scenarios while maintaining compliance.
3Measurement precision
If manual review and human-in-the-loop processes are implemented for data masking, then masking accuracy is improved, but processing time and complexity increase
Solution Approach 1:
Human reviewers provide feedback on automatically masked data, which is then used to refine and improve the automated masking algorithms. This feedback loop allows the system to learn from human expertise and progressively improve masking accuracy while reducing the need for extensive manual review over time.
Solution Approach 2:
The system performs preliminary automated masking before human review, pre-processing the data to handle the majority of masking requirements. This preliminary action reduces the workload for human reviewers and accelerates the overall process while maintaining high accuracy through subsequent human verification of critical cases.
Data Source
AI summary
In some examples, machine learning and rule-based identification, anonymization, and de-anonymization of sensitive structured and unstructured data may include receiving input data that is to be masked, and determining, for the input data, at least one type '1 of entity extraction from a plurality of types of entity extractions to be performed on the input data. The at least one determined type of entity extraction may be performed on the input data, and at least one entity may be extracted from the input data. At least one replacement strategy may be determined from a plurality of replacement strategies for the at least one extracted entity. Further, the at least one determined replacement strategy may be applied to the at least one extracted entity to generate masked data.