Metadata-Based Data Classification for Faster Dataset Discovery
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Conventional scanning tools that utilize AI-based classifiers to identify sensitive data in large datasets consume extensive computing resources and are limited in operation, particularly when dealing with varied data structures, leading to inefficiencies and increased risk of data breaches due to data sprawl.
Innovation Solution
A data discovery system that utilizes a metadata-based recommendation repository to classify data sources by matching metadata from a data source schema with metadata-based recommendations, and employs a classifier model to generate predicted labels for low confidence matches, dynamically updating the repository to enhance applicability and operability across structured and unstructured data.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Measurement precision
If AI-based classifiers are used to scan large datasets to identify sensitive data, then data classification accuracy is improved, but computational resource consumption and processing time increase significantly
Solution Approach 1:
The patent segments the data classification task into two distinct phases: (1) metadata extraction and matching phase that operates on lightweight metadata without scanning actual data content, and (2) AI-based classifier phase that only processes data when metadata matching fails or returns low confidence scores. This segmentation allows the system to handle most classifications efficiently through metadata while maintaining accuracy through selective AI intervention.
Solution Approach 2:
The system performs preliminary actions by extracting and analyzing metadata (such as data source schemas, column names, data types) before attempting to classify actual data content. The metadata-based recommendation repository pre-stores classification rules and patterns that can be quickly matched against incoming data metadata, enabling fast preliminary classification decisions without loading or scanning large volumes of actual data.
2Reliability
If conventional scanning tools scan data structure and content to identify sensitive data, then classification completeness is improved, but system impact and processing time worsen
Solution Approach 1:
The patent extracts and utilizes metadata (data source schemas, column definitions, data types) separate from the actual data content. By taking out the structural information and using it for classification decisions, the system achieves comprehensive classification coverage without needing to scan or load the actual sensitive data content, thereby maintaining reliability while reducing system impact.
Solution Approach 2:
The metadata acts as an intermediary between the scanning system and the actual data content. The metadata-based recommendation repository serves as a mediator that provides classification guidance based on structural information, allowing the system to make classification decisions without directly examining sensitive data content, thus reducing system impact while maintaining classification completeness.
3Measurement precision
If AI-based classifiers are trained on structured data, then classification accuracy for structured data is improved, but applicability to varied data structures deteriorates
Solution Approach 1:
The metadata-based recommendation repository is designed to be universal and adaptable to multiple data structures and formats. It stores classification rules and patterns that can match various metadata schemas (relational databases, NoSQL, data lakes, unstructured data metadata), allowing the same repository to serve diverse data sources. When metadata matching fails, the AI classifier provides fallback classification capability, ensuring versatility across different data structures while maintaining accuracy through the combination of universal metadata matching and specialized AI classification.
Data Source
AI summary
This disclosure describes one or more implementations of systems, non-transitory computer-readable media, and methods that utilize a repository of metadata-based recommendations to classify data sources using metadata from the data sources. For example, the disclosed systems can generate a repository of metadata-based recommendations that indicate recommended classifications for objects within data sources through metadata associated with a data source schema. In some instances, the disclosed systems identify metadata from a data source schema associated with the data source. Subsequently, the disclosed systems can match the identified metadata to a metadata-based recommendation via metadata mappings in the metadata-based recommendation repository to select a metadata-based recommendation. Furthermore, the disclosed systems can also utilize a classifier model to generate predicted labels for the data source and update the metadata-based recommendation repository with a mapping between the predicted labels and metadata corresponding to the data source schema of the data source.