PII Detection Framework for Database Column Classification
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Conventional software solutions for Personally Identifiable Information (PII) detection are inadequate as they focus on text analysis and fail to identify PII at the database level, leading to errors and misclassifications due to subjective human review and the need for predefined rules that do not capture all instances, especially in databases with numerous data columns.
Innovation Solution
A multi-layered PII sensitivity detection framework utilizing machine learning models and rule-based pattern-matching algorithms to analyze column metadata and content, adapting to specific languages and organizational regulations, to accurately identify and mask PII in databases.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Measurement precision
If manual review by human experts is used to identify PII, then the process can be performed with existing tools, but it is prone to errors and misclassifications due to subjective skills and understanding
Solution Approach 1:
The patent replaces the mechanical human review process with an automated machine learning system. The ML model analyzes column metadata, data patterns, and contextual information to identify PII, eliminating subjective human judgment and providing consistent, scalable classification across all database columns without fatigue or variability.
Solution Approach 2:
The system enables the database to self-identify PII through automated analysis of its own metadata and data patterns. The ML model autonomously evaluates column characteristics, data distributions, and contextual relationships to classify PII, reducing dependency on external human reviewers and enabling continuous self-improving classification.
2Productivity
If manual review methods are used to identify PII, then the process can be performed with existing tools, but it is not scalable
Solution Approach 1:
The patent replaces the mechanical human review process with an automated machine learning system. The ML model analyzes column metadata, data patterns, and contextual information to identify PII, eliminating subjective human judgment and providing consistent, scalable classification across all database columns without fatigue or variability.
Solution Approach 2:
The system transforms the identification process by changing from human-centric parameters (subjective expertise, manual effort) to computational parameters (data patterns, metadata analysis, statistical relationships). This enables the system to process vast numbers of columns simultaneously while maintaining or improving accuracy through consistent application of analytical criteria.
3Measurement precision
If conventional software solutions focus on text analysis, then they can detect PII in presented content, but they fail to identify PII at the database level where data is represented differently
Solution Approach 1:
The patent moves the detection approach from the text presentation dimension to the database metadata dimension. By analyzing column metadata, data types, patterns, and contextual relationships at the database level, the system identifies PII before it is presented to users, handling different data representations through structured metadata analysis rather than relying on text context alone.
Solution Approach 2:
The system introduces metadata analysis as an intermediary layer between the raw database data and the final PII classification. By using metadata (column names, data types, patterns, relationships) as a mediator, the system can infer PII presence even when the actual data values are obscured, abbreviated, or presented in non-obvious formats.
4Productivity
If solely rule-based PII detection is used, then predefined rules can be applied systematically, but the rules must be defined by users or system administrators which does not capture all instances of PII when a database has hundreds of data columns
Solution Approach 1:
The system enables the database to self-identify PII through automated analysis of its own metadata and data patterns. The ML model autonomously evaluates column characteristics, data distributions, and contextual relationships to classify PII, reducing dependency on external human reviewers and enabling continuous self-improving classification.
Solution Approach 2:
The system transforms the identification process by changing from human-centric parameters (subjective expertise, manual effort) to computational parameters (data patterns, metadata analysis, statistical relationships). This enables the system to process vast numbers of columns simultaneously while maintaining or improving accuracy through consistent application of analytical criteria.
Data Source
AI summary
Methods and systems herein identify and redact personally identifiable information. A PII sensitivity detection framework includes multiple layers where each layer corresponds to a computer model. The framework analyzes data stored within different data tables and predicts whether a data column includes PII. The first layer corresponds to an artificial intelligence model that analyzes each column metadata and predicts a first score indicative of a likelihood of PII. The second layer corresponds to a rule-based computer model that uses various rules to determine a second score indicative of a likelihood of PII for each column. The third layer corresponds to a column content model that analyzes content of each column using various natural language processing techniques to generate a third score indicative of a likelihood of PII. The framework masks data being presented to a user based on the scores generated via execution of one or more of the layers.