Data Scrubbing System Using Negative Word Index for Sensitive Data Removal
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Organizations face challenges in identifying and removing sensitive data, such as personally identifiable information (PII) and protected health information (PHI), from logged communications like phone calls and online chats, especially when this data is embedded in machine learning models, leading to potential loss of important information and security risks.
Innovation Solution
A data scrubbing system uses a negative word index constructed from publicly available information to identify and remove sensitive data from text transcripts by parsing the text using a sliding window mechanism, matching words and sequences with statistical thresholds to determine non-sensitive data and redact sensitive information.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If the entire transcript is deleted to remove sensitive data, then security requirements are met, but important non-sensitive information is lost
Solution Approach 1:
The patent segments the transcript into individual words and word sequences, allowing selective identification and removal of only sensitive portions while preserving non-sensitive content. The system divides the text into manageable units (words, n-grams) that can be independently evaluated against sensitivity criteria.
Solution Approach 2:
The patent extracts and removes only the sensitive data portions from the transcript while leaving the rest of the content intact. This selective extraction approach removes PII, PHI, and other sensitive information without deleting entire transcripts or useful non-sensitive information.
2Reliability
If the machine learning model is re-trained to remove sensitive data, then sensitive data is completely removed, but all learned information is lost and re-training is required
Solution Approach 1:
The patent applies sensitivity detection and redaction to the training data before it is fed into the machine learning model. By pre-processing the training corpus to remove sensitive information, the model learns from sanitized data without requiring re-training after deployment, saving significant time and computational resources.
Solution Approach 2:
The patent introduces an intermediary data processing layer between the raw training data and the machine learning model. This intermediary scrubbing system filters sensitive information from the training corpus, allowing the model to be trained on clean data without direct exposure to sensitive information, eliminating the need for post-training remediation.
3Ease of manufacture
If traditional keyword-based methods are used to identify sensitive data, then implementation is simple, but accuracy is low and false positives occur
Solution Approach 1:
The patent moves beyond simple keyword matching by incorporating multiple parameters for sensitivity determination, including contextual analysis, statistical frequency thresholds, and confidence scores. The system evaluates words and word sequences against multiple criteria rather than relying on a single keyword list, significantly improving identification accuracy.
Solution Approach 2:
The patent implements feedback mechanisms where the system continuously refines its sensitivity detection based on statistical analysis of identified patterns and confidence scores. The model learns from its own performance and adjusts its identification criteria, reducing false positives while maintaining implementation feasibility through automated iterative improvement.
Data Source
AI summary
Methods and systems are presented for automatically scrubbing sensitive data from text data comprising a sequence of words based on a negative word index. The negative word index may be constructed by obtaining articles that are publicly available and extracting words and word sequences from the articles. Statistical information associated with the word and word sequences from the articles may also be determined and included in the negative word index. To scrub sensitive data from the text data, a first sub-sequence of words is identified from the text data. The first sub-sequence of words may be determined to include sensitive information or not based on statistical information associated with the first sub-sequence of words within the negative word index. If the first sub-sequence of words includes sensitive information, the first sub-sequence of words may be removed from the text data.