Missing Value Imputation with Reason Encoding for ML Accuracy

Resolve Bottlenecks,
Find Innovative SolutionsGenerate Solutions

Solution Overview

Problem

Machine learning models face challenges in managing missing values in datasets, which can lead to biased models and reduced accuracy, especially when the reasons for missing data are not properly accounted for.

Innovation Solution

A computer-implemented system and method for managing missing values in machine learning models by computing data statistics, identifying and verifying missing values, imputing them based on statistical analysis, and encoding the reasons for their absence, allowing users to select imputation methods and parameters, and combining imputed values with encoded reasons for improved model accuracy.

Engineering Contradictions & Design Principles

VSEngineering Contradiction Analysis

1Productivity

If traditional machine learning models are used with incomplete datasets, then model training can proceed quickly, but the model accuracy and reliability deteriorate due to missing values

Engineering Contradiction:

Improvemodel training speedVSAvoidmodel accuracy

Core Design Contradiction:

ProductivityVSReliability

Solution Approach 1:

The system performs preliminary identification and imputation of missing values before model training begins. By pre-processing the dataset to fill missing values using statistical methods (mean, median, mode imputation) or advanced techniques (KNN imputation, regression imputation), the system ensures that the training process can proceed without interruptions while maintaining high model accuracy through properly handled missing data.

Inventive Principle:

Principle #10Preliminary action

2Loss of time

If simple imputation methods are used to fill missing values, then processing time is reduced, but manufacturing precision of data quality deteriorates

Engineering Contradiction:

Improvedata processing timeVSAvoiddata quality

Core Design Contradiction:

Loss of timeVSManufacturing precision

Solution Approach 1:

The system dynamically selects imputation methods based on the characteristics of the missing data and the specific dataset being processed. It offers multiple imputation strategies ranging from simple (mean, median, mode) to advanced (KNN, regression, iterative imputation), allowing users to choose the appropriate level of complexity. This dynamic approach ensures optimal balance between processing time and data quality for different scenarios.

Inventive Principle:

Principle #15Dynamics

Solution Approach 2:

The system changes parameters such as the imputation method type, the number of iterations for iterative imputation, and the distance metric for KNN imputation to optimize the balance between processing speed and data quality. By adjusting these parameters, users can control the trade-off between computational resources consumed and the quality of imputed values produced.

Inventive Principle:

Principle #35Parameter changes

3Device complexity

If missing values are not properly handled, then data processing is simpler, but information loss increases leading to biased models

Engineering Contradiction:

Improvedata processing complexityVSAvoidinformation completeness

Core Design Contradiction:

Device complexityVSLoss of information

Solution Approach 1:

The system introduces an intermediary imputation layer between raw data collection and model training. This intermediary component automatically identifies missing values, determines appropriate imputation strategies, and fills gaps using statistical or machine learning-based methods. By acting as a mediator, it preserves information that would otherwise be lost, prevents model bias, and maintains data integrity without requiring complex manual intervention.

Inventive Principle:

Principle #24Intermediary (Mediator)

4Measurement precision

If manual verification of missing values by experts is performed, then data accuracy improves, but productivity decreases due to time-consuming processes

Engineering Contradiction:

Improvemissing value identification accuracyVSAvoiddata processing throughput

Core Design Contradiction:

Measurement precisionVSProductivity

Solution Approach 1:

The system implements self-service automated identification and verification of missing values using statistical analysis and machine learning algorithms. It automatically detects missing data patterns, verifies their validity through consistency checks, and applies appropriate imputation methods without requiring manual expert intervention. This self-service capability maintains high accuracy in identifying and handling missing values while dramatically increasing data processing throughput compared to manual methods.

Inventive Principle:

Principle #25Self-service

Data Source

PatentUS11875239B2Managing missing values in datasets for machine learning models

Publication Date: 2024.01.16 FAIR ISAAC & CO INC

AI summary

Computer-implemented machines, systems and methods for managing missing values in a dataset for a machine learning model. The method may comprise importing a dataset with missing values; computing data statistics and identifying the missing values; verifying the missing values; updating the missing values; imputing missing values; encoding reasons for why values are missing; combining imputed missing values and the encoded reasons; and recommending models and hyperparameters to handle special or missing values.