ML Feature Preprocessing Using Correlation and Dependency Filtering
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing machine learning models face challenges in reducing complexity and preventing overfitting due to the inclusion of unnecessary features, especially when the learning mechanism is unclear, leading to increased model complexity and the need for large datasets.
Innovation Solution
A data preprocessing method using an autoencoder or Lasso regression model to identify feature influence and generate a sub-feature list, selecting features based on correlation coefficients with a target variable to reduce model complexity and time required for training.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Adaptability or versatility
If features are selected without verifying interrelationship or independence, then the model can be trained with available data, but the model complexity increases and overfitting occurs
Solution Approach 1:
The patent applies preliminary action by performing feature independence verification and selecting independent features before model training. The system calculates correlation coefficients between features and identifies independent feature sets prior to training, thereby preventing model complexity increase and overfitting while enabling effective training with available data
Solution Approach 2:
The patent extracts only the necessary independent features from the feature set by calculating correlation coefficients and removing dependent features. This extraction process reduces the feature set to only those features that are both correlated with the target variable and independent of each other, preventing model complexity increase
2Adaptability or versatility
If unnecessary features are included in training, then the model can be trained with available data, but overfitting occurs
Solution Approach 1:
The patent performs preliminary feature selection by calculating correlation coefficients between features and the target variable, and verifying feature independence before model training. This preliminary action ensures that only relevant and independent features are used, preventing overfitting while maintaining model training capability
Solution Approach 2:
The patent extracts and removes features that are either not correlated with the target variable or are dependent on other features. This extraction process eliminates unnecessary features that would cause overfitting, thereby improving model generalization while preserving training capability with available data
3Loss of information
If the number of features is increased, then the model can capture more information, but the time required for training increases
Solution Approach 1:
The patent extracts only the essential independent features that are correlated with the target variable, removing redundant and dependent features. This extraction maintains the information necessary for accurate modeling while significantly reducing the feature set size, thereby reducing training time without losing critical information
Solution Approach 2:
The patent performs preliminary feature selection and independence verification before model training by calculating correlation coefficients. This preliminary action identifies and retains only the most informative independent features, capturing essential information while minimizing the number of features to reduce training time
Data Source
AI summary
Provided is a method of preprocessing data for efficient machine learning. The method includes generating a feature prediction model based on a training dataset including a plurality of features of a target variable; generating, using the feature prediction model, a sub-feature list, which is a list of other features dependent on each feature constituting the training dataset; calculating correlation coefficients between the plurality of features and the target variable based on the training dataset; and selecting a feature to be used for training a model that predicts the target variable, from among the plurality of features based on the correlation coefficients and the sub-feature list.