Anomaly Detection Model Using Similarity Regularization
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Machine learning models for anomaly detection suffer from poor performance when trained with sparse data sets, leading to false negatives and false positives.
Innovation Solution
The use of similarity regularization, where coefficients for a machine learning model with a sparse data set are determined by leveraging coefficients from similar entities with dense data sets.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Quantity of substance
If a machine learning model is trained with a sparse data set, then the training process can be completed with limited data, but the model performance suffers with increased false negatives and false positives
Solution Approach 1:
The patent combines data from multiple related entities to train a unified machine learning model. Instead of training separate models for each entity with sparse data, the system merges datasets across entities (e.g., multiple retailers, products, or time periods) to create a larger combined training set, thereby improving model reliability while maintaining applicability to individual entities
Solution Approach 2:
The patent develops a universal machine learning model that can be applied across multiple entities and contexts. The model is designed to be entity-agnostic, learning patterns that generalize across different retailers, products, or time periods, allowing a single model to serve multiple functions and entities simultaneously
2Reliability
If a machine learning model is trained with a dense data set, then the model performance improves with reduced false negatives and false positives, but the requirement for large amounts of training data increases
Solution Approach 1:
The patent segments the overall anomaly detection problem into entity-specific and shared components. By identifying and separating entity-agnostic patterns from entity-specific characteristics, the system can train on large combined datasets for shared patterns while maintaining effectiveness for individual entities, reducing the data burden for each specific application
3Reliability
If similarity regularization is applied to leverage coefficients from similar entities, then the model performance improves with sparse data, but the complexity of determining entity similarity increases
Solution Approach 1:
The patent transforms the similarity assessment problem by changing the parameters used to evaluate entity similarity. Instead of complex multi-dimensional comparisons, the system uses simplified parameters such as data characteristics, temporal relationships, and structural properties to determine entity similarity, reducing computational complexity while maintaining effectiveness
Data Source
AI summary
Disclosed herein are embodiments for anomaly detection using machine learning models (MLMs) and similarity regularization. An embodiment operates by obtaining data for a first product, a second product, and a target product. The data include a set of sparse data points for the target product. Next, similarity scores between the target product and the first product and the second product may be calculated. The embodiment further operates by generating a target MLM associated with the target product using a regularization penalty. The regularization penalty is based on the similarity scores and distances between a target set of coefficients for the target MLM and coefficients for a first MLM and a second MLM associated with the first product and the second product, respectively. The embodiment may then detect an anomaly associated with the target product by feeding the target MLM with a feature vector associated with the target product.