Database Target Data Detection Through Iterative Subsampling
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing data management systems face inefficiencies and resource-intensive challenges in identifying sensitive information within databases, particularly due to intrusive full table scans that impact production environments.
Innovation Solution
An iterative subsampling technique is employed to identify the storage locations of target data, adjusting sample sizes based on table dimensions and using randomized seeds to generate representative subsamples, discarding columns without target data, and iteratively refining the sampling process for accuracy and efficiency.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Measurement precision
If full table scans are used to identify sensitive information, then detection completeness is improved, but system performance and resource utilization deteriorate
Solution Approach 1:
The patent segments the database table into multiple subsets (partitions) and performs detection on each subset separately. This allows the detection process to be distributed and parallelized, improving system performance while maintaining detection completeness across the entire table.
Solution Approach 2:
The patent uses sampling techniques to detect sensitive information in a subset of rows rather than scanning the entire table. By strategically selecting representative samples and using confidence scoring, the system achieves adequate detection coverage without the full resource cost of a complete table scan.
2Measurement precision
If full table scans are performed, then detection accuracy is improved, but resource consumption increases
Solution Approach 1:
The system performs detection on a sampled subset of data rather than the complete table. Confidence scores are calculated based on the sampled data, and thresholds are applied to determine sensitive information presence. This partial action approach maintains acceptable detection accuracy while significantly reducing computational resource consumption.
Solution Approach 2:
The patent changes the parameter of detection scope from 100% of table rows to a configurable sample percentage. By adjusting this parameter, the system can balance between detection accuracy and resource consumption based on specific operational requirements.
3Productivity
If iterative subsampling is used to reduce computational load, then processing efficiency is improved, but detection time may increase
Solution Approach 1:
The detection process is segmented into multiple iterative passes, each working on progressively refined subsets of data. Columns are evaluated and removed from subsequent passes when confidence thresholds are met, reducing the data volume in later iterations and improving overall processing efficiency despite multiple passes.
Solution Approach 2:
The system performs preliminary detection passes on sample data to establish confidence scores and identify columns containing sensitive information. These preliminary results guide subsequent full-table detection efforts, allowing the system to focus computational resources only on relevant columns and reduce overall detection time.
Data Source
AI summary
Methods, systems, and devices for data management are described. The method may include obtaining first subsamples of a data table, where the first subsamples include information from a first quantity of columns of the data table. The method may also include processing the first subsamples of the data table to identify whether the information included in the first subsamples includes a target type of information. The method may include obtaining a second subsample of the data table, where the second subsample includes information from a subset of columns of the first quantity of columns. The method may include processing the second subsample of the data table to identify whether information included in the subset comprises the target type of information and identifying, based at least in part on processing the one or more first subsamples and the second subsample, one or more locations of the target type of information within the data table.