Non-Relational Data Classification Under Memory Constraints
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing data classification systems struggle to efficiently detect and classify sensitive information in non-relational databases due to memory limitations and the lack of structured metadata, making it difficult to comply with data protection regulations like GDPR and CCPA.
Innovation Solution
A non-relational data classification system partitions the database into manageable segments, iteratively analyzes each document for metadata and data values, populates a structured table, and uses data classification techniques to detect sensitive information, while clearing memory for the next partition, and auto-populates missing metadata with null values.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If the entire non-relational database is loaded into memory for analysis, then complete data can be examined, but memory limitations prevent this from being feasible for large databases
Solution Approach 1:
The database is partitioned into multiple smaller partitions that can be processed iteratively. Each partition is loaded into memory, analyzed for sensitive information, and then discarded, allowing complete database scanning without requiring the entire database to reside in memory simultaneously.
2Adaptability or versatility
If non-relational databases are analyzed without structured metadata, then flexibility is maintained, but data classification becomes difficult due to lack of structured information
Solution Approach 1:
A structured metadata table is created as an intermediary representation of the non-relational database documents. This table transforms the unstructured document format into a structured format with fields for metadata and data values, enabling systematic analysis while preserving the original flexible data structure.
3Quantity of substance
If iterative analysis of partitions is performed, then memory constraints are respected, but processing time increases compared to single-pass analysis
Solution Approach 1:
The system continuously iterates through partitions in a systematic manner, maintaining the same analysis process for each partition. This continuous iterative approach ensures complete coverage while managing memory constraints, accepting the time cost as necessary for thorough analysis of large datasets.
4Reliability
If sensitive information detection is performed on all data, then comprehensive security coverage is achieved, but system complexity increases
Solution Approach 1:
The system automatically detects and classifies sensitive information without requiring manual configuration or complex rule sets. The classification model self-adjusts to identify sensitive data patterns, reducing the need for complex manual setup while achieving comprehensive security coverage.
Data Source
AI summary
A method, a computer program product, and a system for detecting sensitive information in a non-relational database. The method includes partitioning a non-relational database into a plurality of partitions that are based characteristics relating to the non-relational database. The method also includes analyzing, by proceeding in an iterative manner with a first partition from the plurality of partitions, detecting a plurality of documents within the first partition containing metadata and data values, populating a table with the metadata representing fields and the data values relating to the metadata representing records, detecting sensitive information in the table, and recording the sensitive information. The method further includes proceeding with a second partition acting as a next entry for a next iteration, until all partitions in the plurality of partitions are analyzed, and reporting the sensitive information gathered from the partitions.