PII Detection Pipeline With Feedback-Driven RegEx Refinement
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing computer-implemented techniques for detecting personally identifiable information (PII) in large corpora of documents are inefficient, costly, and impractical due to high computational demands, especially when context analysis is required, making it difficult to comply with regulatory obligations.
Innovation Solution
Implementing a chain of successively more accurate PII detectors where downstream stages inform upstream stages, utilizing regular expression (RegEx) detection and machine learning, with feedback mechanisms to improve upstream detector accuracy and reduce computational costs.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Measurement precision
If accurate PII detection is performed using context analysis and machine learning, then detection accuracy is improved, but computational cost and processing time increase significantly
Solution Approach 1:
The detection system is divided into multiple sequential stages: a first detector using computationally efficient methods (regex, keyword matching) followed by a second detector using more accurate but computationally intensive methods (machine learning, context analysis). This segmentation allows the system to achieve high accuracy only when necessary, reducing overall computational cost while maintaining detection precision.
2Reliability
If comprehensive PII detection is performed across large corpora of documents, then detection completeness is improved, but processing time and resource consumption increase
Solution Approach 1:
The system processes documents through a multi-stage pipeline where the first detector quickly screens all documents to identify potential PII candidates, and the second detector performs comprehensive analysis only on those candidates. This ensures detection completeness for all documents while maintaining high processing throughput by avoiding intensive analysis on non-PII content.
Solution Approach 2:
The system applies full computational power (context analysis, machine learning) selectively only to portions of the corpus that are likely to contain PII, rather than applying it uniformly to all documents. This partial action approach maintains detection completeness for relevant documents while significantly improving overall processing productivity.
3Speed
If simple detection methods are used, then processing speed is improved, but detection accuracy decreases
Solution Approach 1:
The detection process is segmented into two speed-accuracy tiers: the first detector provides fast initial screening with acceptable accuracy, and the second detector provides slow but highly accurate verification for ambiguous cases. This segmentation allows the system to achieve high overall accuracy while maintaining fast processing for clear-cut cases.
Data Source
AI summary
Methods, systems, and computer program products for detection of personally identifiable information (PII). A first detector and a second detector are configured to interoperate. The first detector is different from the second detector and the second detector incurs a greater computational cost than the first detector when processing identical content. Content is presented to the first detector so as to implement a first type of PII detection that is based at least in part on regular expression analysis using regular expressions. The content is presented to the second detector. The second detector performs PII detection based on content analysis that is different from the first detector's regular expression analysis. The second detector causes generation of new regular expressions based on the content analysis and the first detector is updated with such new regular expressions. Performance of the first detector is continually improved as new regular expressions are generated.