Missing Value Imputation via Iterative Model Refinement
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing methods for imputing missing values in data sets often result in decreased performance, especially when most rows have missing values filled with the mean of non-missing values, leading to loss of data efficacy in machine learning applications.
Innovation Solution
A system and method that initializes missing values using most likely values based on frequency distribution for categorical data and mean/variance for continuous data, generates mathematical models for each column, updates these models iteratively until an error metric converges, and fills missing values using these models, allowing for repeated updates and error determination using RMSE.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If missing values are filled with the mean of non-missing values, then the data set can be completed for machine learning applications, but the performance of machine learning applications decreases due to loss of data efficacy
Solution Approach 1:
The patent segments the imputation process into multiple iterations, where in each iteration, mathematical models are generated for columns with missing values based on non-missing values, then used to update missing values. This segmented approach allows the system to progressively refine imputations rather than using a single mean value for all missing entries, thereby maintaining data efficacy while improving imputation accuracy.
Solution Approach 2:
The patent implements feedback through iterative updating of mathematical models based on the imputed values. In each iteration, the models are regenerated and refined based on the updated data set, allowing the system to learn from previous imputations and improve subsequent ones. This feedback mechanism ensures that imputation accuracy increases while preserving data efficacy for machine learning applications.
2Measurement precision
If mathematical models are generated and iteratively updated for each column with missing values, then imputation accuracy improves, but the complexity of the data processing system increases
Solution Approach 1:
The patent employs universal mathematical models that can be applied across multiple columns with missing values. Instead of creating specialized imputation methods for each column, the system uses a unified approach where the same type of mathematical models (e.g., regression models) are generated and updated for all relevant columns. This universality reduces system complexity while maintaining high imputation accuracy.
Solution Approach 2:
The patent merges the imputation process for multiple columns into a single iterative framework. Rather than processing each column independently through separate complex procedures, the system combines all column imputations into one unified iterative process where models are generated, updated, and applied across the entire data set simultaneously. This merging reduces overall system complexity while achieving accurate imputations.
Data Source
AI summary
A computer readable medium includes a data set with data stored in rows and N columns. Each of the rows is associated with one individual patient. Each of the N columns is associated with one type of data for patients. One or more processors is configured to: initialize missing values in M ones of the N columns with M values for the M ones of the N columns, respectively; generate M mathematical models for the M ones of the N columns having one or more missing values; for each of the rows having one or more missing values, update ones of the M values for the M ones of the N columns; and fill missing values in the M ones of the N columns with the M values, respectively.