Imputing Missing Values in Machine Learning via Iterative Partitioning
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing machine learning models often discard data with missing feature values, leading to performance distortion and underestimation of standard deviation, especially in datasets with heterogeneity, due to assumptions like MCAR and MNAR that do not always hold.
Innovation Solution
A method involving partitioning datasets into subsets with and without missing values, using a predictive model to generate residual values for imputation, and iteratively refining the model by swapping training and verification datasets to accurately fill in missing values while reducing noise.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Quantity of substance
If missing values are replaced with mean, median, or mode under MCAR assumption, then the data can be used for modeling, but the distribution is distorted and standard deviation is underestimated
Solution Approach 1:
The patent segments the dataset into multiple subsets based on the presence or absence of missing values for different features. By partitioning data into groups (e.g., records with complete values vs. records with missing values for specific features), the method enables targeted imputation strategies for each segment, preserving the heterogeneity of the data and avoiding uniform distortion across the entire dataset.
Solution Approach 2:
The patent changes the imputation parameter from static values (mean, median, mode) to dynamic predicted values generated by machine learning models. The imputed value for each missing feature is determined by training a model on records with complete values and using that model to predict missing values in records with gaps, thereby adapting the imputation to the specific context of each data point rather than applying a uniform approach.
2Reliability
If data with missing values is discarded until threshold is met, then model performance is maintained, but data quantity and utilization are reduced
Solution Approach 1:
The patent performs preliminary imputation of missing values using trained machine learning models before the data is used for modeling. By pre-filling missing values with predicted values based on patterns learned from complete records, the method prepares the data in advance to meet modeling requirements without discarding records, thus maintaining both data quantity and model reliability.
Solution Approach 2:
The patent employs iterative training where models are trained on complete records, used to impute missing values, and then the process is repeated with updated data. This feedback loop continuously improves the quality of imputed values and the performance of the models, allowing progressive utilization of previously unusable data while maintaining reliability standards.
3Loss of information
If constant value is used for imputation under MNAR assumption, then missingness pattern is accounted for, but heterogeneity in data records is ignored
Solution Approach 1:
The patent applies local quality by training separate machine learning models for different features and different data segments rather than using a single constant value for all missing values. Each model is specialized to predict values for specific features based on patterns in the data, thereby accounting for local characteristics and heterogeneity in different parts of the dataset while still addressing the MNAR assumption through context-aware prediction.
Data Source
AI summary
In a machine learning environment, missing values can be imputed based upon an expectation maximization style approach. A system partitions a dataset, and uses a first partition as a training subset and a second partition as a verification subset. The training subset is used to train a machine learning model, which is then used to impute missing values in the second subset. The subsets may be swapped and the process iterates to predict missing values in the dataset with a high degree of accuracy, thereby improving both the accuracy of the machine learning model and the accuracy of the imputed values. The noise in the value prediction is reduced through a linear regression setting to account for heterogeneity in the dataset.