Covariance Matrix Estimation Using Missing Rate Weighting
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Building a statistical model from high-dimensional data with many missing values is computationally intensive and reduces accuracy, and existing methods for estimating covariance matrices deteriorate when data contains many missing values.
Innovation Solution
An information processing apparatus that calculates the missing rate and estimates a covariance matrix based on this rate, minimizing the distance to an initial estimate while considering the missing rate, thereby maintaining estimation accuracy even with high missing rates.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If missing value imputation is performed to build a statistical model from high-dimensional data, then the model can be constructed, but the amount of calculation becomes huge and accuracy deteriorates when there are many missing values or large number of dimensions
Solution Approach 1:
The patent extracts and processes only the necessary components for covariance matrix estimation without performing full missing value imputation. By calculating missing rates and using weighted estimation formulas, it extracts the essential statistical information needed for model construction while avoiding the computational burden of imputing all missing values in high-dimensional data.
2Measurement precision
If existing covariance matrix estimation methods are used when data contains many missing values, then the estimation can be performed, but the estimation accuracy of the covariance matrix deteriorates
Solution Approach 1:
The patent applies local quality by introducing weighted estimation where different elements of the covariance matrix are estimated with different weights based on their missing rates. Elements with lower missing rates receive higher weights, while those with higher missing rates receive lower weights. This localized weighting strategy maintains estimation accuracy by adapting to the specific quality of each data element rather than applying a uniform approach.
Solution Approach 2:
The patent changes the estimation parameter by introducing missing rate as a weighting factor in the covariance matrix estimation formula. Instead of using standard estimation methods that treat all data points equally, the patent modifies the estimation parameter to account for varying data quality, thereby improving estimation accuracy under high missing rate conditions.
3Ease of manufacture
If standard covariance matrix estimation is performed without considering missing rate, then the process is simple, but the accuracy of the statistical model deteriorates when data contains many missing values
Solution Approach 1:
The patent performs preliminary action by calculating the missing rate for each feature before performing covariance matrix estimation. This preliminary step allows the subsequent estimation process to be weighted appropriately, ensuring that the simplicity of the overall process is maintained while accuracy is improved through the pre-computed missing rate information.
Data Source
AI summary
An information processing apparatus includes a data acquisition unit that acquires data including a missing value, a missing rate calculation unit that calculates a missing rate indicating a ratio of missing values included in the data, and a covariance matrix estimation unit that estimates a covariance matrix based on the missing rate. According to the information processing apparatus, since the covariance matrix is estimated based on the missing rate, the estimation accuracy of the covariance matrix can be improved.