Iterative Database Subsampling for Sensitive Data Detection
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing data management systems face challenges in efficiently identifying and locating sensitive information within databases without disrupting production environments, as traditional full table scans are resource-intensive and impact read/write operations.
Innovation Solution
An iterative subsampling technique is employed to identify the presence of target data in databases, where columns are progressively dropped based on initial analysis, using randomized seeds and adjusted sample sizes to ensure representative coverage and reduce computational load.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Measurement precision
If full table scans are used to identify sensitive information in databases, then detection completeness is improved, but system performance and productivity deteriorate due to resource intensity and impact on read/write operations
Solution Approach 1:
The patent segments the database scanning process into multiple iterations, each scanning a subset of tables or columns rather than the entire database at once. This segmentation allows the system to maintain detection completeness while reducing the resource burden on any single scan operation, thereby preserving system performance during production operations.
Solution Approach 2:
The patent implements partial scanning by selecting and scanning only specific subsets of database tables or columns in each iteration, rather than performing exhaustive full table scans. This partial action approach maintains adequate detection coverage while significantly reducing computational resources consumed and minimizing impact on read/write operations.
2Productivity
If iterative subsampling with column dropping is implemented, then computational overhead is reduced, but detection precision may worsen due to sampled data representation
Solution Approach 1:
The patent incorporates feedback mechanisms where results from each scanning iteration inform the selection and configuration of subsequent iterations. The system analyzes detected patterns, adjusts sampling strategies, and refines column selection based on previous findings, thereby maintaining detection accuracy while improving computational efficiency through adaptive subsampling.
Solution Approach 2:
The patent performs preliminary analysis in early iterations to identify and eliminate columns or tables that are unlikely to contain sensitive information. This preliminary action allows subsequent iterations to focus computational resources on high-priority areas, maintaining detection precision while reducing overall computational overhead through intelligent pruning.
Data Source
AI summary
Methods, systems, and devices for data management are described. The method may include obtaining first subsamples of a data table, where the first subsamples include information from a first quantity of columns of the data table. The method may also include processing the first subsamples of the data table to identify whether the information included in the first subsamples includes a target type of information. The method may include obtaining a second subsample of the data table, where the second subsample includes information from a subset of columns of the first quantity of columns. The method may include processing the second subsample of the data table to identify whether information included in the subset comprises the target type of information and identifying, based at least in part on processing the one or more first subsamples and the second subsample, one or more locations of the target type of information within the data table.