Data Classification Using Clustering and Neighboring Sets
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Conventional automated data classification techniques fail to classify columns in data sets effectively, leading to unclassified data that can pose security risks and compliance issues due to missing classifications.
Innovation Solution
The method involves using machine learning to determine clusters of columns in previously classified data sets, identifying missing columns, and classifying them by analyzing neighboring data sets, thereby creating new data classes and improving data governance and security.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Measurement precision
If conventional automated data classification techniques are used, then the classification process is simple and fast, but the classification accuracy is low and many columns remain unclassified
Solution Approach 1:
The system performs preliminary classification of columns using existing classification techniques before applying cluster-based classification. This two-stage approach allows the system to leverage both simple conventional methods and more complex cluster-based methods, improving overall classification accuracy while maintaining efficiency for easily classifiable columns.
Solution Approach 2:
The system introduces cluster identification as an intermediary step between conventional classification and final classification results. By identifying clusters of related columns and using cluster context to inform classification decisions, the system bridges the gap between simple individual column classification and complex holistic classification, achieving better accuracy without fully committing to complex systems.
2Measurement precision
If cluster-based classification is applied to improve classification accuracy, then more columns are classified correctly, but the processing time and computational resources increase
Solution Approach 1:
The system applies cluster-based classification selectively rather than to all columns. It focuses cluster-based methods on columns that are difficult to classify using conventional techniques or that belong to identified clusters, while leaving easily classifiable columns to be processed by faster conventional methods. This partial application reduces overall processing time while maintaining high accuracy for critical columns.
3Reliability
If comprehensive data classification is performed to ensure compliance with EILM rules, then security and compliance are improved, but the system complexity and resource requirements increase
Solution Approach 1:
The system segments the data classification problem into distinct components: conventional classification of individual columns, cluster identification based on column relationships, and cluster-based classification. This segmentation allows each component to be optimized independently and implemented modularly, improving compliance reliability through comprehensive classification while managing system complexity through structured decomposition.
Data Source
AI summary
A method of data classification includes: identifying a cluster of data classes; classifying columns of a current data set; identifying the cluster in the current data set; determining, based on the cluster, an expected column is missing from the current data set; determining a neighboring data set; identifying the expected column in the neighboring data set; classifying the expected column in the neighboring data set; creating a new data class in the current data set; and classifying an unclassified column in the current data set or the neighboring data set with the new data class.