Column Classification Model Using Scoring Models
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Conventional data platforms face inefficiencies in classifying columns of tables due to the need for manual review and predefined category rules that do not scale well with large datasets, leading to errors and inefficiencies.
Innovation Solution
The system intelligently classifies columns by accessing their features, applying scoring models to generate confidence values for category assignments, allowing users to define new classification rules and models, such as machine learning models, to automate the classification process.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Productivity
If manual review and predefined category rules are used for column classification, then classification accuracy can be maintained for small datasets, but the process becomes inefficient and does not scale well with large datasets
Solution Approach 1:
The system enables columns to classify themselves by automatically generating and applying scoring models to column features. The classification process is self-driven through confidence value calculations and threshold comparisons, eliminating the need for manual review while maintaining consistency and accuracy across large datasets.
Solution Approach 2:
The system transforms the classification approach by changing from static predefined rules to dynamic scoring models that calculate confidence values. This parameter change allows the system to adapt to different column types and characteristics, improving both scalability and accuracy simultaneously.
2Adaptability or versatility
If predefined category rules are used for classification, then the system is simple to implement, but it cannot adapt to new classification needs and does not scale well
Solution Approach 1:
The system introduces dynamic scoring models that can be selectively applied to different column categories. The classification process adapts to new requirements by generating confidence values based on column features and applying appropriate scoring models, providing flexibility without requiring complete system redesign.
Solution Approach 2:
The classification system is segmented into multiple scoring models, each targeting specific column categories. This segmentation allows the system to handle different classification needs independently, improving adaptability while keeping each individual scoring model relatively simple and manageable.
3Reliability
If manual classification review is performed, then classification accuracy can be verified, but it consumes significant time and human resources
Solution Approach 1:
The system implements automated feedback through confidence value calculations and threshold-based verification. Each column classification is automatically verified by comparing confidence values against thresholds, providing reliable verification without manual intervention. The system can also identify low-confidence classifications for selective human review, optimizing the balance between automation and verification.
Data Source
AI summary
Systems and methods for classifying columns using a model are provided. The systems and methods access a table associated with a column of features and retrieve a list of categories each associated with a different scoring model. The systems and methods, for each category in the list of categories, apply a respective scoring model to the features of the column to generate a respective set of confidence values indicating a likelihood that the column belongs to a respective one of the categories. The systems and methods process the respective sets of confidence values to select a target category from the list of categories and associate the selected target category with the column.