Data Complementing Method Using Correlation-Based Regression
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing methods for complementing missing data in machine learning, such as MI and FIML, often result in reduced accuracy when the missing ratio is high, leading to low accuracy in regression and statistical complements, and may require removing records with missing values, thereby reducing the number of learning records.
Innovation Solution
A data complementing method that calculates the correlation matrix between attributes, performs regression complementation for attributes with high correlation and statistical complementation for those with low missing ratios, and deletes records with uncomplemented missing values, adjusting thresholds to balance accuracy and record quantity.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Measurement precision
If regression complement is used for missing values, then accuracy of complement is improved when high correlation attributes exist, but accuracy deteriorates when no high correlation attribute is available
Solution Approach 1:
The system dynamically changes the complement method based on the correlation parameter between attributes. When correlation exceeds a threshold, regression complement is applied; otherwise, statistic complement is used. This parameter-based selection resolves the contradiction by adapting the method to the data characteristics.
Solution Approach 2:
The complement strategy is made dynamic by calculating correlation coefficients and selectively applying different complement methods (regression or statistic) based on the calculated correlation strength. This dynamic adaptation ensures high accuracy when correlation exists while maintaining reliability when it does not.
2Ease of manufacture
If statistic complement is used for missing values, then complement process is simplified, but accuracy deteriorates when missing ratio is high
Solution Approach 1:
The system uses the missing ratio as a parameter to determine the complement method. When missing ratio is below a threshold, statistic complement is applied for simplicity; when it exceeds the threshold, regression complement is used to maintain accuracy. This resolves the contradiction between simplicity and accuracy.
Solution Approach 2:
The complement approach dynamically switches between statistic and regression methods based on the missing ratio of each attribute. This dynamic selection ensures process simplicity when missing ratios are low while maintaining high accuracy when missing ratios are high.
3Reliability
If records with missing values are removed, then data quality is improved, but the number of learning records decreases
Solution Approach 1:
The system segments the handling of missing values by attribute, calculating correlation and missing ratios separately for each attribute. This allows selective complementation per attribute rather than removing entire records, preserving data quantity while improving quality through targeted complementation.
Solution Approach 2:
The complement method is applied locally to each attribute based on its specific correlation characteristics and missing ratio, rather than applying a global rule to all records. This local quality approach maintains data quality by complementing only where necessary while preserving the overall dataset.
4Measurement precision
If multiple imputation method is used, then both attribute values in record and same attribute of other records are utilized, but calculation complexity increases
Solution Approach 1:
The system segments the complex MI calculation into simpler, attribute-by-attribute processing. By calculating correlation and missing ratios for each attribute separately and applying appropriate complement methods, the system achieves MI-like accuracy without the full computational complexity of traditional multiple imputation.
Solution Approach 2:
The system extracts the essential benefit of MI (utilizing both within-record and across-record information) while removing the computational burden by using correlation-based selective complementation instead of full multiple imputation modeling.
Data Source
AI summary
A data complementing method implemented by a computer, the method includes: calculating degree of correlation of a data item corresponding to a missing data value with another data item, in a case where the missing data value exists in a plurality of data records including data values corresponding to a plurality of data items, respectively; performing complementation of the missing data value by a recursive method based on a data item value of the other data item, in a case where the degree of correlation is larger than a predetermined correlation threshold; and performing complementation of the missing data value by a statistical method based on a data value other than the missing data value of a data item corresponding to the missing data value, in a case where the degree of correlation is not larger than the correlation threshold.