Perturbed Records Generation via Correlation Matrix Feature Reduction
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Machine learning models generate a large number of perturbed records, which can be excessive and inefficient, due to the lack of effective methods to reduce the number of features represented in these records, leading to increased computational resources and scoring requests.
Innovation Solution
The method involves generating a correlation matrix to identify highly correlated data feature pairs, removing one feature from each pair, and using the reduced dataset to create perturbed records, thereby reducing the number of features and records, and identifying arbitrary indicators associated with proposed actions.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Measurement precision
If all features are included in perturbed records generation, then comprehensive bias detection is achieved, but the number of perturbed records becomes excessively large
Solution Approach 1:
The patent extracts and removes highly correlated features from the feature set before generating perturbed records. By identifying features with correlation coefficients above a threshold (e.g., 0.95) and removing redundant ones, the system reduces the number of features that need to be perturbed, thereby significantly decreasing the total number of perturbed records generated while maintaining bias detection effectiveness.
Solution Approach 2:
The patent implements a nested structure where feature correlation analysis is performed within the perturbed records generation process. The correlation matrix computation and feature pair identification are nested within the overall bias detection workflow, allowing for efficient reduction of the feature space before perturbation operations are applied.
2Reliability
If comprehensive feature perturbation is performed, then thorough model testing is achieved, but computational resources increase significantly
Solution Approach 1:
The patent extracts highly correlated feature pairs using correlation matrix computation and removes one feature from each pair before perturbation. This extraction process identifies and eliminates redundant computational work, ensuring that perturbation operations are only performed on independent features, thereby reducing computational resource consumption while maintaining testing thoroughness.
Solution Approach 2:
The patent changes the parameter set by transforming the original feature space into a reduced feature space based on correlation analysis. By modifying which features are included in the perturbation process (removing highly correlated ones), the system optimizes computational efficiency without compromising the reliability of model testing.
3Loss of information
If redundant correlated features are included, then complete data representation is maintained, but the scoring endpoint footprint increases
Solution Approach 1:
The patent extracts and removes redundant highly correlated features from the dataset before generating perturbed records. By computing correlation matrices and identifying feature pairs with correlation coefficients above a threshold, the system eliminates duplicate information representation, thereby reducing the scoring endpoint footprint while maintaining complete data representation through the remaining independent features.
Data Source
AI summary
Reducing a count of perturbed records in a machine learning dataset by application of a correlation matrix of feature values identified in training records to reduce the number of features represented in the perturbed records. Deleting one of a pair of correlated records is achieved with reference to a correlation score that identifies features of sufficient similarity to be paired up. Reducing the number of features for which values are assigned in a data perturbation process results in a relatively reduced number of perturbed records.