Context Pattern Matching for Accurate Sensitive Data Labeling
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing systems struggle to accurately identify sensitive data due to variations in context and misspellings, leading to mislabeling, and lack the ability to adapt to rapid changes in context-based rules.
Innovation Solution
A context-aware pattern matching system that uses inclusive and exclusive regex patterns, combined with machine learning models, to accurately label sensitive data by generating and updating patterns based on feedback, ensuring that candidate tokens are correctly associated with data labels.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Productivity
If rules-based systems use regular expression patterns to capture sensitive data, then detection speed is improved, but detection accuracy deteriorates due to misspellings and variations
Solution Approach 1:
The system implements feedback loops where detection results are continuously evaluated and used to refine regex patterns. Misclassified tokens are fed back into the system to update patterns, enabling the rules-based system to learn from errors and improve accuracy while maintaining detection speed through automated pattern generation.
Solution Approach 2:
The regex patterns are made dynamic rather than static. The system automatically generates and updates patterns based on detected variations in sensitive data formats. This allows the detection rules to adapt to new misspellings and variations without manual intervention, resolving the contradiction between fast rule-based detection and accurate coverage of variations.
2Adaptability or versatility
If the system updates patterns frequently to account for context changes, then adaptability is improved, but system complexity increases
Solution Approach 1:
The system performs self-updating of patterns without requiring external intervention. It automatically detects context changes, generates new patterns, and refines existing ones based on feedback from detection results. This self-service capability enables frequent adaptation to context changes while keeping the system relatively simple by eliminating the need for manual pattern management infrastructure.
Solution Approach 2:
The system prepares and updates patterns in advance based on anticipated context changes and feedback from ongoing operations. By proactively refining patterns before they are needed for detection, the system maintains high adaptability while managing complexity through automated preliminary updates rather than reactive complex modifications.
3Measurement precision
If the system uses context analysis to determine data labels, then labeling accuracy is improved, but processing time increases
Solution Approach 1:
The context analysis process is segmented into discrete, manageable components that can be processed efficiently. The system divides context into relevant segments around candidate tokens and applies pattern matching to each segment independently. This segmentation enables accurate context-based labeling while reducing overall processing time through parallelizable, modular analysis steps.
Data Source
AI summary
A method includes generating first patterns indicating a data label and associating a candidate token of a text sequence with the data label by removing first tokens from the text sequence based on a match of the first tokens with a token of second patterns and selecting the candidate token from other tokens of the text sequence based on a match between the candidate token and a token of the second patterns. The method also includes updating a token sequence collection to comprise the candidate token and a context token, updating the second patterns with new patterns that match the candidate token and the context token, and removing a first pattern from the second patterns based on a determination that the first pattern matches with a token sequence associated with the test tokens.