Statistical Sensitive Data Detection via ML Inference
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Detecting sensitive data within large data sets in cloud computing environments is challenging due to the impracticality of examining all contents, especially in secure environments, and existing methods often result in high false positives or fail to differentiate between sensitive and non-sensitive information.
Innovation Solution
Employing statistical techniques that utilize external data sets and machine learning models to infer the presence of sensitive data by analyzing statistical relationships, reducing data through hierarchical arrangement and semantics, and applying obfuscation to ensure privacy, with simple models like logistic regression for rapid training and execution.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Measurement precision
If all contents of large data sets are examined to detect sensitive data, then detection accuracy is improved, but computational resources and time are excessively consumed
Solution Approach 1:
The patent segments the large data set into multiple partitions or blocks, and processes each partition independently using distributed computing frameworks. This allows the system to examine content with high accuracy while dividing the computational burden into manageable chunks that can be processed in parallel, reducing overall computation time and resource consumption.
Solution Approach 2:
The patent employs sampling techniques where a representative subset of the data is examined in detail, while other portions are analyzed using statistical methods or heuristic rules. This partial examination approach maintains acceptable detection accuracy without requiring exhaustive analysis of every single data point, thereby significantly reducing computational time.
2Measurement precision
If traditional data analysis methods are used to detect sensitive data, then comprehensive detection is achieved, but false positive rates increase
Solution Approach 1:
The patent implements feedback mechanisms where the system continuously learns from detection results, adjusting its parameters and thresholds based on observed patterns. By incorporating feedback loops that analyze false positives and refine detection rules, the system maintains high detection accuracy while progressively reducing false positive rates through iterative improvement.
Solution Approach 2:
The patent dynamically adjusts detection parameters such as sensitivity thresholds, confidence levels, and matching criteria based on the specific data being analyzed and the type of sensitive information being sought. This adaptive parameter adjustment allows the system to optimize the balance between detection accuracy and false positive reduction for different data contexts.
3Measurement precision
If complex machine learning models are used for sensitive data detection, then detection capability is improved, but training time and computational overhead increase
Solution Approach 1:
The patent segments the machine learning model into multiple smaller sub-models or feature extractors that can be trained independently on different data partitions. This segmentation allows parallel training of model components, reducing overall training time while maintaining the comprehensive detection capability that would require a single large complex model.
Solution Approach 2:
The patent introduces intermediary processing layers or feature extraction stages that transform raw data into simplified representations before feeding them to the main detection model. These intermediaries reduce the dimensionality and complexity of the input data, allowing the model to achieve high detection capability with reduced training requirements and computational overhead.
Data Source
AI summary
A candidate attribute combination of a first data set is identified, such that the candidate attribute combination meets a data type similarity criterion with respect to a collection of data types of sensitive information for which the first data set is to be analyzed. A collection of input features is generated for a machine learning model from the candidate attribute combination, including at least one feature indicative of a statistical relationship between the values of the candidate attribute combination and a second data set. An indication of a predicted probability of a presence of sensitive information in the first data set is obtained using the machine learning model.