Anomaly Detection Feature Matrix Manipulation
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Machine learning models for unsupervised anomaly detection face challenges in high-dimensional datasets due to the curse of dimensionality, leading to sparse training datasets that result in sub-optimal detection performance, especially for isolated or low-incidence anomalies.
Innovation Solution
A computer-implemented method and system that receives data on entity usage, extracts feature vectors, generates a feature matrix, assigns binary values based on a usage threshold, identifies k nearest neighbor entities, and modifies usage values based on their usage and distance, resulting in a manipulated feature matrix to optimize feature spaces for anomaly detection models.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Loss of information
If the entire feature set is used to construct the training dataset for unsupervised anomaly detection, then relevant anomaly information is retained, but the training dataset becomes sparse and detection sensitivity decreases
Solution Approach 1:
The patent extracts and removes zero-variance features (features where all training samples have the same value) from the feature set before training the anomaly detection model. This extraction process eliminates features that contribute to sparsity without providing discriminative information, thereby retaining only the useful anomaly-related features in the training dataset.
Solution Approach 2:
The patent changes the parameter configuration by identifying and filtering features based on their variance properties. By computing the variance of each feature across training samples and removing those with zero variance, the patent transforms the feature set to achieve optimal balance between information retention and detection sensitivity.
2Device complexity
If dimensionality reduction techniques are applied, then the data representation becomes compact, but isolated anomaly features may be lost
Solution Approach 1:
The patent extracts and removes zero-variance features that would be lost in traditional dimensionality reduction. By identifying features with zero variance across training samples and explicitly removing them before model training, the patent preserves potentially important isolated anomaly features while still achieving dimensionality reduction through the removal of non-informative features.
Solution Approach 2:
The patent changes the approach from traditional dimensionality reduction (which transforms features) to feature selection based on variance analysis. By computing variance as a parameter and using it to filter features, the patent achieves compact representation while preserving isolated anomaly features that would otherwise be lost.
3Adaptability or versatility
If distance-based anomaly detection models are trained on high-dimensional sparse datasets, then the model can handle complex features, but the model becomes insufficiently sensitive to detect low-incidence behavior
Solution Approach 1:
The patent extracts and removes zero-variance features from high-dimensional datasets before training distance-based anomaly detection models. This extraction eliminates features that increase dimensionality without providing discriminative power, thereby reducing sparsity while maintaining the model's ability to handle complex anomaly patterns.
Solution Approach 2:
The patent changes the dataset configuration by filtering features based on variance parameters. By removing features with zero variance, the patent transforms high-dimensional sparse datasets into lower-dimensional denser datasets, improving the sensitivity of distance-based models to low-incidence anomalies while preserving adaptability to complex feature patterns.
Data Source
AI summary
A method including: receiving a set of data representing usage by entities of objects in a computing resource; extracting, from the initial set of data, one or more feature vectors representing the usage by one of the entities with respect to the objects; generating, from the feature vectors, a feature matrix; with respect to each entry in the feature matrix: (i) assigning a binary value to the entry, based on a predefined usage threshold, (ii) identifying, among the one or more entities, k nearest neighbor entities with respect to the one of the entities, based on a predefined distance threshold, and (iii) modifying the usage value of the entry, based on usage values associated with each of the k nearest neighbor entities with respect to the one of the objects; and updating the feature matrix with the modified usage values, to obtain a manipulated feature matrix.