PII Detection via Rule-Based Filtering and LLM Verification
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Conventional systems fail to accurately detect Personally Identifiable Information (PII) in documents due to limitations in Regular Expression tools, particularly with formats varying by country and issues with Optical Character Recognition, leading to false negatives and inefficiencies.
Innovation Solution
A method that combines deterministic rule-based searches with Large Language Models (LLMs) for Named Entity Recognition (NER), utilizing a three-stage process involving initial deterministic searches, LLM training, and selective LLM-based analysis to enhance PII detection accuracy and efficiency, while maintaining data security by processing documents locally.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Measurement precision
If deterministic rule-based search is used for PII detection, then the system is simple and fast, but the detection accuracy is low due to format variations and OCR issues
Solution Approach 1:
The system segments the PII detection process into two distinct stages: a deterministic rule-based search stage for initial detection and a probabilistic LLM-based verification stage for accurate classification. This segmentation allows each stage to specialize - rules handle speed and simple cases while LLM handles accuracy and complex cases, resolving the contradiction between simplicity and accuracy.
Solution Approach 2:
The system dynamically adjusts the detection approach based on document characteristics. Documents are first evaluated by deterministic rules, and only those flagged as potential matches proceed to the more complex LLM-based analysis. This dynamic filtering optimizes resource usage while maintaining high detection accuracy.
2Measurement precision
If probabilistic LLM-based search is used for PII detection, then the detection accuracy is high, but the processing time and computational resources increase
Solution Approach 1:
The system applies the computationally intensive LLM-based analysis only partially - specifically to documents that are flagged by the initial deterministic rule-based search. This partial application of the more powerful method avoids the excessive processing time that would result from applying it to all documents, while still achieving high accuracy for relevant cases.
Solution Approach 2:
The deterministic rule-based search serves as a preliminary filtering stage that quickly identifies candidate documents before they undergo the more time-consuming LLM-based analysis. This preliminary action reduces the volume of documents requiring intensive processing, thereby reducing overall processing time while maintaining accuracy.
3Measurement precision
If LLM is trained on extracted snippets from documents, then the model becomes more accurate for NER, but data security risks increase due to potential data exposure
Solution Approach 1:
The system extracts only the minimal necessary textual snippets surrounding detected PII data-items for LLM training, rather than using entire documents. This extraction principle reduces the amount of sensitive data exposed during the training process while still providing sufficient context for the LLM to learn accurate NER patterns, thereby balancing accuracy improvement with data security.
4Productivity
If conventional Regular Expression tools are used, then the system is efficient and fast, but false negatives increase due to format variations by country
Solution Approach 1:
The LLM-based NER system acts as an intermediary layer between the deterministic rule-based search and the final PII identification. It mediates by verifying and correcting detections from the rule-based system, particularly for cases with varied international formats, thereby improving reliability without significantly impacting the overall efficiency provided by the initial fast filtering.
Data Source
AI summary
Device, system, and method for automatically detecting and classifying personally identifiable information (PII) in documents and files. A method includes performing a deterministic rule-based search, in a plurality of stored documents, for PII data-items. If the deterministic rule-based search indicates that a particular document is more likely than not to contain a PII data-items then the method includes: extracting a textual snippet from the particular document, wherein the textual snippets surrounds the PII data-item; adding the textual snippet and the particular document to one or more training datasets utilized for training a Large Language Model (LLM) configured to find PII data-items in documents for Named Entity Recognition (NER) in those documents.