Missing Data Imputation Using Regression Trees and Correlation Analysis
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing methods for handling missing parameter values in data sets, such as average imputation and weighted mean techniques, fail to accurately capture variations and are inadequate for sudden changes, particularly in scenarios with network issues or faulty sensors.
Innovation Solution
The method employs completeness and correlation information across dependent parameters to build dynamic decision trees and uses auto encoders to refine estimates, combining predicted values with global and local averages using a complementary filter.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If average imputation is used to fill missing parameter values, then the data set can be completed, but the method fails to capture variations and sudden changes in parameter values
Solution Approach 1:
The patent segments the data set into multiple groups based on similarity metrics, and applies different imputation strategies to each group. Instead of using a single global average, the method creates localized groups where missing values are imputed based on group-specific statistics, thereby capturing variations while maintaining completeness.
Solution Approach 2:
The patent applies local quality by using group-specific average values rather than a single global average for all data points. Each group receives tailored imputation based on its own statistical properties, allowing the method to capture local variations and sudden changes in parameter values while maintaining overall data completeness.
2Measurement precision
If more data entries are used to improve estimation accuracy, then the representation of parameter variations improves, but noise and computational complexity increase
Solution Approach 1:
The patent reduces computational complexity by segmenting the data into groups and performing imputation at the group level rather than considering all individual data points. This segmentation allows the method to use more data for estimation while managing computational load through efficient group-based processing.
Solution Approach 2:
The patent uses group average values that represent multiple underlying data points, effectively using more information than a single value would provide. This partial aggregation approach allows the method to incorporate excessive data entries for improved accuracy while the group averaging process inherently filters out some noise.
Data Source
AI summary
According to an aspect, there is provided a computer-implemented method for processing a data set, the data set comprising respective data subsets for a plurality of subjects, each data subset comprising a plurality of data entries, each entry comprising respective parameter values for each of a plurality of parameters at a respective time point, wherein for a first data subset relating to a first subject in the plurality of subjects, one or more parameter values for at least a first parameter in the plurality of parameters is missing from the first data subset, the method comprising, for a first missing parameter value in a first data entry in the first data subset (a) determining completeness scores for the first parameter, wherein each completeness score indicates a level of completeness of the data entries in the first data subset for the first parameter and a respective one of the other parameters in the plurality of parameters; (b) determining correlation scores for the first parameter, wherein each correlation score indicates a level of correlation between the parameter values in the data set for the first parameter and the parameter values in the data set for a respective one of the other parameters in the plurality of parameters; (c) determining a subset of the plurality of parameters to use to form regression trees based on the determined completeness scores and the determined correlation scores; (d) forming a plurality of regression trees, wherein each regression tree relates to a respective parameter combination of the first parameter and one or more of the other parameters in the determined subset, and each regression tree is trained to predict a parameter value for the first parameter based on input parameter values for the one or more other parameters in the parameter combination, wherein each regression tree is trained using training data comprising parameter values for the parameters in the respective parameter combination, wherein the training data includes the parameter values in any data entry in the first data subset for which a parameter value is present for all of the parameters in the respective parameter combination; (e) using each regression tree to predict a parameter value for the first parameter based on parameter values in the first data entry for the one or more other parameters in the parameter combination; and (f) combining the predicted parameter values to estimate the first missing parameter value. A corresponding apparatus and computer program product are also provided.