Multivariate Outlier Removal Using Hotelling's T2 and DModX
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Conventional multivariate modeling techniques are hindered by manual and time-consuming outlier removal processes, leading to inconsistent and degraded model performance due to the presence of outliers in historical data.
Innovation Solution
Automated detection and removal of outliers using both univariate and multivariate statistical techniques, involving interquartile range filtering and calculation of Hotelling's T2 and DModX values to refine historical data before training multivariate models.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If manual outlier removal is used, then model robustness can be improved, but time consumption and operational complexity increase significantly
Solution Approach 1:
The system performs preliminary automated outlier detection and removal using univariate and multivariate statistical techniques before model training begins. This preliminary action eliminates the need for time-consuming manual outlier review while ensuring robust training data, directly resolving the contradiction between model robustness and time consumption.
Solution Approach 2:
The system enables self-service automated outlier removal through multiple statistical methods (univariate filtering, Hotelling's T2, DModX) that automatically identify and remove outliers without requiring manual intervention. This self-service capability maintains model robustness while dramatically reducing the time and operational effort required compared to manual processes.
2Reliability
If manual outlier removal is used, then model robustness can be improved, but operational complexity and standardization increase
Solution Approach 1:
The outlier removal process is segmented into distinct automated stages: univariate filtering using interquartile ranges, multivariate analysis using Hotelling's T2 and DModX statistics, and iterative refinement. This segmentation creates a standardized, reproducible workflow that improves model robustness while reducing operational complexity through automation and clear process definition.
Solution Approach 2:
The system automatically adjusts statistical parameters (confidence levels, threshold values, iteration counts) to optimize outlier detection while maintaining model robustness. This parameter automation eliminates the complexity of manual parameter tuning and standardizes the outlier removal process across different datasets and models.
3Reliability
If comprehensive outlier removal is performed, then model performance consistency is improved, but computational resources and processing time increase
Solution Approach 1:
The system applies partial outlier removal by focusing computational resources on the most impactful outlier detection methods (Hotelling's T2 and DModX) rather than exhaustively applying all possible statistical techniques. This partial action achieves sufficient model performance consistency while controlling computational resource usage and processing time.
Solution Approach 2:
Univariate filtering using interquartile ranges is performed as a preliminary step before multivariate analysis, quickly eliminating obvious outliers and reducing the dataset size. This preliminary action improves model performance consistency while minimizing the subsequent computational resources required for more intensive multivariate outlier detection.
Data Source
AI summary
In a method for improving multivariate model performance, a first data set comprising values of a plurality of features and corresponding labels is obtained. A second data set is generated from the first data set. Generating the first data set includes generating an intermediate data set by removing a first set of outliers from the first data set using a univariate statistical technique, generating a first multivariate model using the intermediate data set, and removing a second set of outliers from the first data set using the first multivariate model and a multivariate statistical technique. A second multivariate model is generated using the second data set.