PII Detection via Cross-Dataset Frequency Analysis
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing methods for detecting and protecting personally identifiable information (PII) in big data collections are inadequate due to reliance on conventional techniques like dictionaries and regular expressions, which struggle with context-dependent identification, varied formats, and errors such as spelling mistakes, and cannot distinguish between personal and business-related data.
Innovation Solution
A statistical analysis method that compares the frequency of data elements in user-specific datasets to multi-user datasets to identify and protect PII, using tokenization, frequency analysis, filtering, and machine learning to predict and proactively protect sensitive information without language specificity, complementing conventional PII recognizers.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Measurement precision
If conventional PII recognizers using dictionaries and regular expressions are used, then known PII formats can be identified, but they fail to handle context-dependent identification, varied formats, spelling errors, and cannot distinguish personal from business-related data
Solution Approach 1:
The patent changes the detection parameter from static pattern matching to dynamic frequency analysis. By computing frequency ratios of tokens across different datasets (user-specific vs. multi-user), the system adapts to varied PII formats, spelling variations, and contextual differences without requiring pre-defined dictionaries or rules.
Solution Approach 2:
The patent replaces the mechanical system of regular expression matching with a statistical machine learning approach. Instead of relying on predefined patterns and dictionaries, the system uses frequency analysis and probability calculations to identify PII, enabling it to handle unseen formats and contextual variations effectively.
2Reliability
If targeted encryption of known sensitive data fields is applied, then specific PII fields can be protected, but free form text fields containing PII cannot be effectively protected
Solution Approach 1:
The system enables free form text fields to protect themselves automatically through frequency-based detection. Instead of requiring manual specification of which fields to encrypt, the system analyzes token frequencies across datasets to automatically identify PII within free form fields and apply appropriate protection, making the protection mechanism self-adapting to any field type.
Solution Approach 2:
The patent creates a universal PII protection mechanism that works across all field types (structured and free form) using the same frequency analysis approach. The system doesn't require different protection strategies for different field types, making the solution universally applicable to any data structure or format.
3Reliability
If encryption of the entirety of each data collection is applied, then PII is effectively protected, but the data collection becomes much harder to use for legitimate uses
Solution Approach 1:
The patent applies protection selectively to only those data elements identified as PII based on frequency analysis, rather than encrypting the entire dataset. This localized approach ensures that only sensitive information is protected while leaving the rest of the data collection accessible and usable for legitimate purposes, maintaining both security and functionality.
4Reliability
If validation of every bit of information in user inputs is performed, then PII input can be prevented, but the process becomes cumbersome and unpractical
Solution Approach 1:
The system performs preliminary frequency analysis on data elements to identify potential PII before final data processing or storage. By pre-computing frequency ratios and identifying suspicious tokens in advance, the system can flag or protect PII automatically without requiring cumbersome real-time validation of every user input, reducing complexity while maintaining reliability.
Data Source
AI summary
Certain aspects of the present disclosure provide techniques for detecting and protecting personally identifiable information. In one example, a method includes retrieving a user-specific dataset from a multi-user dataset; filtering the user-specific dataset to create a user-specific data subset; determining a user frequency of each user-specific token of a plurality of user-specific tokens in the user-specific data subset; determining a multi-user frequency for each user-specific token of the plurality of user-specific tokens in the multi-user dataset; computing a frequency ratio based on the user-specific frequency and the multi-user frequency of each user-specific token of the plurality of user-specific tokens; and protecting each user-specific token whose frequency ratio is above a frequency ratio threshold.