Dataset Feature Type Inference for ML Model Training
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Machine learning models are often trained on datasets that lack diversity and representation, leading to inaccurate predictions due to insufficient feature type identification and adjustment.
Innovation Solution
A system and method for feature type inference and dataset adjustment using a feature type inference module and a data adjustment module, which identifies and labels feature types, and enhances the dataset with additional features generated by a large language model to improve model training.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Measurement precision
If traditional dataset training is used without feature type inference, then the training process is simpler and faster, but the prediction accuracy and model reliability deteriorate due to insufficient feature type identification
Solution Approach 1:
The system performs feature type inference and dataset adjustment before the main model training process. By pre-processing the dataset to identify feature types and generate synthetic samples, the system prepares optimized training data in advance, which improves prediction accuracy without adding complexity to the core training algorithm.
Solution Approach 2:
The feature type inference module acts as an intermediary between the raw dataset and the model training process. It analyzes data subsets, identifies feature types, and generates adjusted datasets that bridge the gap between unprocessed data and model-ready training sets, thereby improving accuracy while keeping the training system relatively simple.
2Reliability
If diverse and representative training datasets are created through feature type inference, then model reliability and prediction validity improve, but the data processing time and computational resources increase
Solution Approach 1:
The system divides the dataset into multiple data subsets and processes each subset independently to identify specific feature types. This segmentation allows parallel processing of different data portions, reducing overall processing time while maintaining comprehensive feature type identification across the entire dataset for improved model reliability.
Solution Approach 2:
The system generates synthetic data samples by copying and transforming existing data patterns. Instead of manually creating diverse training examples, the system automatically generates synthetic copies that preserve feature type characteristics, thereby expanding dataset diversity and improving model reliability without proportionally increasing processing time.
3Measurement precision
If multiple feature type candidates are evaluated with different machine learning models, then the selection of optimal feature types improves accuracy, but the computational complexity and processing time increase
Solution Approach 1:
The system evaluates multiple feature type candidates using different machine learning models, but applies this comprehensive evaluation only to critical data subsets or high-priority features. By performing partial exhaustive analysis on key features and using simpler methods for less critical ones, the system maintains high feature type identification accuracy while preserving overall processing efficiency.
Data Source
AI summary
According to an aspect of an embodiment, one or more operations may include accessing a dataset including multiple data subsets. Feature type candidates corresponding to the data subsets may be identified. The one or more operations may further include building first machine learning models using different sets of feature type candidates. Each of the different sets of feature type candidates may be scored based on respective accuracies, relative to the dataset, of each first machine learning model that respectively corresponds to each different set of feature type candidates. A final set of feature types may be selected from the different sets of feature type candidates based on the scores of the different sets of feature types. The operations may further include training a second machine learning model using a labeled dataset that is generated by applying the final set of feature types to the dataset.