Bi-local linear model for missing value imputation in correlated datasets
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Conventional missing value imputation methods in databases do not effectively consider co-related data features, which can impact the accuracy of predictions in datasets like personal health records, where features may be linearly correlated.
Innovation Solution
A bi-local linear model is employed to partition records into groups based on co-related data types, such as quantitative and qualitative data, and use these partitions to predict missing values by generating weighted linear functions for each type of data, combining them to estimate the missing value.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Measurement precision
If conventional missing value imputation methods are used that treat all features as independent, then the method is simple to implement, but the prediction accuracy deteriorates when data features are linearly correlated
Solution Approach 1:
The patent segments the dataset into multiple subsets based on data types (e.g., quantitative data, qualitative data, time-series data). Each subset is processed separately with appropriate imputation methods, allowing the system to handle correlated features within each segment while maintaining overall accuracy. This resolves the contradiction by dividing the complex problem into manageable parts.
Solution Approach 2:
The patent dynamically changes imputation parameters based on data characteristics. When linear correlation is detected among features, the system switches from simple mean imputation to correlation-aware methods like regression-based imputation or KNN imputation. This adaptive parameter adjustment maintains high accuracy without permanently increasing system complexity.
2Reliability
If all data features are treated as independent in imputation, then the processing speed is fast, but the quality of imputation deteriorates for correlated data
Solution Approach 1:
The patent implements a dynamic imputation system that adapts its processing approach based on the correlation structure of the data. The system first performs a quick independence check, then dynamically selects whether to apply simple or complex imputation methods for each feature subset. This dynamic adaptation ensures high imputation quality for correlated data while maintaining fast processing for independent features.
Solution Approach 2:
The patent applies sophisticated imputation methods only to the extent necessary - specifically to feature subsets where linear correlation is detected. For the majority of independent features, simple fast methods are used. This partial application of complex methods maintains imputation quality where needed while preserving overall processing speed.
3Measurement precision
If conventional imputation methods are used without considering data type categories, then the implementation is straightforward, but the imputation accuracy deteriorates for mixed data types
Solution Approach 1:
The patent segments the dataset into multiple subsets based on data types (e.g., quantitative data, qualitative data, time-series data). Each subset is processed separately with appropriate imputation methods, allowing the system to handle correlated features within each segment while maintaining overall accuracy. This resolves the contradiction by dividing the complex problem into manageable parts.
Solution Approach 2:
The patent creates a universal imputation framework that can handle multiple data types through a common interface. The system provides a unified API that automatically detects data types and applies appropriate methods, making the complex multi-type handling as easy to use as simple imputation while achieving high accuracy across all data types.
Data Source
AI summary
The method includes selecting a target record from a dataset, the target record including a missing value, partitioning records of the dataset into at least two groups including co-related data, the partitioned records including records having a value for a same field as the missing value in the target record, predicting the missing value based on a relationship between fields in each of the at least two groups associated with the partitioned records, and setting the missing value of the target record to the predicted value.