Intelligent Data Sampling Using Recency and Lineage for PII Detection
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing systems fail to efficiently identify and safeguard personally identifiable information (PII) across diverse data structures due to reliance on keyword searches, which are inadequate for detecting non-standard forms of PII, leading to potential exposure and inadequate protection.
Innovation Solution
Implementing intelligent sampling techniques using recency and lineage criteria to narrow down data search parameters, identifying PII by determining when data was updated and its hierarchical relation to known PI structures, followed by applying security policies.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Measurement precision
If keyword search methods are used to identify PII, then the implementation is simple and fast, but the detection accuracy is insufficient and cannot identify non-standard forms of PII
Solution Approach 1:
The patent segments the PII detection process into multiple stages: initial keyword filtering, followed by machine learning-based classification, and finally expert review for borderline cases. This segmentation allows the system to achieve high detection accuracy through layered analysis rather than relying on a single complex method, thus resolving the contradiction between accuracy and complexity.
Solution Approach 2:
The patent introduces an intermediary machine learning classification layer between simple keyword matching and final PII identification. This intermediary uses trained models to analyze contextual patterns and relationships in data, enabling accurate detection of non-standard PII forms without requiring the entire system to be overly complex.
2Reliability
If comprehensive keyword searching is performed across all data, then all PII may be identified, but the computational resources and time required are excessive
Solution Approach 1:
The patent applies preliminary action by performing keyword-based filtering and basic pattern matching before conducting more resource-intensive machine learning analysis. This preliminary screening reduces the data volume that requires comprehensive analysis, thereby maintaining reliable PII identification while significantly improving detection efficiency and reducing computational overhead.
Solution Approach 2:
The patent implements a tiered detection approach where not all data receives the same level of analysis. High-risk data patterns receive full machine learning analysis, while low-risk data receives simpler processing. This partial application of comprehensive analysis maintains protection reliability for critical data while improving overall productivity by avoiding excessive resource consumption on low-priority data.
3Adaptability or versatility
If manual identification and tagging of PII is performed, then the accuracy for known PII types is high, but the process is time-consuming and cannot scale to new PII forms
Solution Approach 1:
The patent implements feedback mechanisms where detected PII patterns and expert review outcomes are used to continuously retrain and improve the machine learning models. This feedback loop enables the system to adapt to new PII forms automatically over time, increasing versatility without requiring manual reconfiguration for each new pattern, while maintaining high detection accuracy.
Solution Approach 2:
The machine learning models perform self-service by automatically learning and adapting to new PII patterns from training data and feedback, reducing the need for manual intervention. This self-learning capability enables the system to scale to new PII forms without proportionally increasing the time investment required for detection.
Data Source
AI summary
A computer-implemented method includes determining that a specific type of information is to be identified in a set of data. The method further includes sampling the set of data according to various sampling criteria to identify the specified type of information. The sampling criteria include at least a recency criterion indicating that the data to be sampled has been updated within a specified timeframe and a lineage criterion indicating that the data to be sampled is within a maximum hierarchical distance from a source data structure. The method also includes identifying, from the data that was sampled according to the sampling criteria, one or more data structures that include the specified type of information. The method further includes applying security policies to the identified data structures based on the type of information that was identified in the set of data. Various other methods, systems, and computer-readable media are also disclosed.