Explanatory Analysis for Missing Data Records
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing machine learning and artificial intelligence systems face challenges in handling incomplete data records due to missing values, which can negatively impact model training and prediction accuracy, as current imputation methods provide varying results and lack explanatory context for the substituted values.
Innovation Solution
A system and method that determine missing values in data records by training a model with complete data records, generating candidate predictors, and selecting those with high importance values to insert into incomplete records, thereby creating tentative records that are fitted into the model to identify the best fit values with explanatory analysis.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Quantity of substance
If manual imputation methods are used to fill missing data values, then the completeness of data records is improved, but the accuracy and reliability of the imputed values cannot be guaranteed
Solution Approach 1:
The patent introduces a machine learning model as an intermediary between the missing data and the imputation process. The model is trained on complete data records and then used to generate candidate predictors for missing values, providing a systematic and accurate method for data completion that goes beyond simple manual imputation techniques
Solution Approach 2:
The patent replaces manual imputation mechanisms with an automated machine learning-based system. The model automatically generates candidate predictors, ranks them by importance, and selects the best fit values, eliminating the need for manual intervention and providing consistent, reproducible results with measurable accuracy
2Quantity of substance
If existing imputation methods are applied to complete missing values, then data completeness is improved, but explanatory context for the substituted values is lost
Solution Approach 1:
The patent implements a feedback mechanism where the model evaluates candidate predictors by injecting them into the trained model and measuring fit values. This feedback loop provides explanatory context by quantifying how well each candidate predictor fits the model, allowing selection of the most appropriate imputed values based on model performance
Solution Approach 2:
The patent changes the approach from static imputation to dynamic parameter-based selection. By calculating predictor importance values and fit values as parameters, the system can objectively compare and select candidate predictors, providing explanatory context through these quantitative measures rather than arbitrary substitution
3Measurement precision
If multiple candidate predictors are generated for missing values, then the quality of imputed values can be improved, but the complexity of the imputation process increases
Solution Approach 1:
The patent segments the imputation process into distinct stages: generating multiple candidate predictors, calculating importance values for each candidate, ranking candidates based on importance, and selecting the best fit. This segmentation makes the complex process manageable and systematic, allowing quality improvement through multiple candidates without overwhelming complexity
4Measurement precision
If predictor importance values are calculated and thresholding is applied, then the selection of candidate predictors becomes more accurate, but the processing time and computational resources increase
Solution Approach 1:
The patent applies partial action by using thresholding to select only the top candidate predictors that exceed a certain importance value threshold. Rather than evaluating all possible candidates equally, the system focuses computational resources on promising candidates, achieving accurate selection while reducing processing time through selective evaluation
Data Source
AI summary
Embodiments relate to a system, computer program product, and method for determining missing values in respective data records with an explanatory analysis to provide a context of the determined values. Such method includes receiving a dataset including incomplete data records that are missing predictors and complete data records. A model is trained with the complete data records and candidate predictors for the missing predictors are generated. A predictor importance value is generated for each candidate predictor and the candidate predictors that have a predictor importance value in excess of a first threshold value are promoted. Respective promoted candidate predictors are inserted into the respective incomplete data records, thereby creating tentative data records. The tentative data records are injected into the model, a fit value is determined for each of the tentative data records, and a tentative data record with a fit value exceeding a second threshold value is selected.