Database Sensitive Data Classification via Sampling and ML
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Current data classification technologies face challenges in efficiently and accurately identifying sensitive data in large databases, particularly in identifying repetitive and redundant data types, which is crucial for data security and compliance with regulations such as GDPR and HIPAA.
Innovation Solution
A method involving the retrieval of database metadata, sampling of record values, generation of probable associations between columns, application of machine learning classifiers, and scoring of field name pairs to identify likely sensitive data fields, using techniques like Shell sort and Bloom filters for repetitiveness measurement and natural-language understanding for similar field names.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Measurement precision
If traditional data classification technologies are used to scan large databases, then comprehensive data coverage is achieved, but processing time and computational resources increase significantly
Solution Approach 1:
The patent segments the database into sampled records and uses the catalog metadata to infer characteristics of unsampled records. By classifying only a sample of records (e.g., 100-1000 samples) and extending classifications to entire tables/columns based on metadata patterns, the system achieves comprehensive coverage without processing every single record, thus reducing processing time while maintaining accuracy.
Solution Approach 2:
The patent performs preliminary classification on sampled records and uses the results to pre-determine classifications for entire tables and columns through metadata analysis. This preliminary action on a subset allows the system to make informed inferences about the entire database structure without exhaustive scanning, significantly reducing processing time while maintaining classification accuracy.
2Reliability
If exhaustive scanning of all database records is performed, then complete data identification is achieved, but computational resources and processing complexity increase
Solution Approach 1:
The patent divides the database into sampled and unsampled portions, using catalog metadata to represent the unsampled data. By segmenting the processing scope to only analyze sampled records while inferring characteristics from metadata, the system maintains identification completeness without the complexity of processing every single record in the entire database.
Solution Approach 2:
The patent introduces catalog metadata as an intermediary that bridges sampled and unsampled data. The metadata acts as a mediator that allows the system to infer characteristics of unsampled records based on sampled record patterns, eliminating the need for exhaustive scanning while maintaining complete data identification capability.
3Productivity
If sampling of record values is performed, then processing efficiency is improved, but classification accuracy may be compromised
Solution Approach 1:
The patent performs preliminary classification on a strategically selected sample of records and uses the results to establish classification patterns that are then applied to entire tables and columns. This preliminary action on a representative sample, combined with metadata validation, ensures that classification accuracy is maintained while processing efficiency is significantly improved.
Solution Approach 2:
The patent incorporates feedback mechanisms where classification results from sampled records are validated against catalog metadata and used to refine the classification model. This feedback loop ensures that the sample-based classifications remain accurate by continuously comparing and adjusting based on metadata patterns, thereby maintaining high accuracy despite the reduced scope of sampling.
Data Source
AI summary
Automated classification of sensitive data in a database, which includes: Retrieving a catalog of a database. Sampling record values from at least some of the columns. Generating a map of probable associations between different columns of tables of the database. Applying a machine learning classifier to the sampled record values, to classify the columns of the sampled records into multiple data classes, some being sensitive data classes. Classifying columns of non-sampled record values according to the classification of the sampled record values, based on the map. Searching all objects of the database for existence of record values of the classified columns, to output value and field name pairs. Scoring the pairs according to a measure of their repetitiveness in the output. Increasing the score of the pairs whose field names are similar. Based on the scores, indicating which fields of the database are likely to include sensitive data.