Feature Recommendation System Handling Missing Data Values
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing methods for extracting features from data with missing values are limited, as they either delete data, leading to incomplete representation, or impute values, affecting analysis accuracy, and rely on single algorithms, resulting in poor accuracy and overfitting.
Innovation Solution
An apparatus and method that convert missing values into constants or random values, apply algorithms like mRMR, JMIM, and ICAP to filter redundant features, and use random forest for importance extraction, providing feature recommendations that are not sensitive to imputation and consider multiple algorithms for improved accuracy.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Device complexity
If data with missing values is deleted, then the analysis process is simplified, but the feature representation becomes incomplete and the method is sensitive to correction results
Solution Approach 1:
The patent introduces an intermediary imputation process that fills missing values with estimated values before feature extraction, acting as a mediator between the raw data with missing values and the feature extraction process. This allows the system to maintain complete feature representation while keeping the analysis process manageable by preprocessing the data.
2Productivity
If missing values are imputed with estimated values, then the feature extraction can proceed, but the analysis accuracy deteriorates because the imputation method does not consider the ratio of missing values
Solution Approach 1:
The patent changes the parameter of missing value handling by introducing the ratio of missing values as a critical factor. Instead of using a fixed imputation method, the system adjusts the imputation strategy based on the proportion of missing values in each feature, thereby maintaining analysis accuracy while enabling feature extraction to proceed.
Solution Approach 2:
The patent makes the imputation process dynamic by adapting the imputation method and intensity based on the missing value ratio. For features with low missing value ratios, minimal imputation is applied, while for features with high missing value ratios, more robust imputation methods are used, allowing the system to optimize accuracy for each feature individually.
3Device complexity
If a single algorithm is used for feature extraction, then the process is simple, but the accuracy is poor and overfitting occurs
Solution Approach 1:
The patent merges multiple feature extraction algorithms into an ensemble approach, combining the results of different algorithms to produce the final feature selection. This integration allows the system to leverage the strengths of each individual algorithm while mitigating their weaknesses, thereby improving accuracy and reducing overfitting without making the process excessively complex.
Solution Approach 2:
The patent implements a feedback mechanism where the performance of each algorithm is evaluated and used to weight or adjust the contribution of that algorithm in the ensemble. This feedback loop allows the system to dynamically optimize the combination of algorithms based on their actual performance, improving feature extraction accuracy while maintaining manageable complexity.
Data Source
AI summary
A method and apparatus that recommends a feature from a data set. The method includes performing a first conversion on the data set including one target variable and a plurality of feature variables by converting a missing value for each of the feature variables into a preset constant, executing a first algorithm to determine a level of redundancy of the plurality of feature variables, using the data set converted by the first conversion and producing first recommendation information that includes a predetermined number of feature variables selected based on a result of the executing the first algorithm.