Hierarchical Clustering Imputation for Missing Data
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Current data imputation methods for missing values in data analytics often result in information loss and are not suitable for various use cases, failing to effectively handle both normalized and non-normalized data.
Innovation Solution
A computer-implemented method that uses hierarchical clustering to identify the closest record to a missing data record, determines a mean estimate based on the cluster, and adjusts the observed value of the closest record based on record similarity to impute the missing field, thereby creating a more complete dataset for analysis.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Productivity
If listwise deletion is used to handle missing data, then the data processing is simple and fast, but important information is lost
Solution Approach 1:
The patent extracts and separates the missing value imputation task from the overall data processing workflow. By identifying missing values and applying targeted imputation methods (mean, median, mode, or custom functions) only to those specific fields, the system preserves all available information while maintaining processing efficiency. This extraction approach avoids the information loss of listwise deletion by keeping complete records and only filling in specific missing components.
2Loss of information
If classical methods (mean, median) are used to impute missing values, then information loss is avoided, but the imputation accuracy is insufficient for complex use cases
Solution Approach 1:
The patent implements a dynamic imputation system that adapts to different data types, missingness patterns, and use cases. Instead of applying a static mean or median to all missing values, the system dynamically selects appropriate imputation methods based on the specific characteristics of each field and dataset. This dynamic approach significantly improves imputation accuracy while preserving information, as it can handle both simple and complex scenarios appropriately.
Solution Approach 2:
The patent allows changing imputation parameters and methods based on data characteristics. Users can specify custom imputation functions, select from multiple methods (mean, median, mode, regression), and adjust parameters to optimize for different scenarios. This flexibility enables the system to maintain high imputation accuracy across diverse use cases while preserving all original information.
3Measurement precision
If existing imputation methods (MICE, missForest) are applied, then imputation accuracy improves for certain cases, but they are not suitable for many use cases and have limitations
Solution Approach 1:
The patent creates a universal imputation framework that can handle a wide variety of use cases through multiple imputation methods and custom functions. The system is designed to work with different data types, missingness patterns, and analytical requirements, making it applicable to diverse scenarios including but not limited to what MICE and missForest handle. This multi-functionality achieves both high accuracy for specific cases and broad applicability across many use cases.
Data Source
AI summary
Imputation of missing data can be provided for both normalized and non-normalized data. Data can be partitioned, and data complete records used to impute values for data missing records. Hierarchical clustering can be used to cluster the data. Imputation can rely on a closed record in a same cluster. Record similarity can be used to adjust an observed value, and an estimated mean can also be incorporated. Useful for augmenting datasets that can be applied to analytics, machine learning, and other scenarios.