Twin Neural Network Anomaly Detection for Rare Event Classification
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Conventional anomaly detection techniques struggle with accurately identifying rare events due to limited training data, leading to high false negative rates and inefficiencies in class balancing methods such as oversampling and undersampling, especially in detecting gray market transactions.
Innovation Solution
A twin machine learning model architecture using identical feedforward neural networks with shared weights to generate embeddings for data samples, calculating similarity scores, and employing a Sigmoid activation function to classify data records as anomalous, with a data sampling module to generate diverse training pairs for few-shot learning.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If conventional anomaly detection techniques are used with limited training data, then the detection system can operate with available data, but the false negative rate increases and detection accuracy deteriorates
Solution Approach 1:
The patent creates synthetic copies of rare anomalous events by generating additional training samples through data augmentation techniques. This allows the model to learn from more examples of rare events without requiring actual additional real-world anomalous data, directly addressing the limitation of scarce training data for rare event detection
Solution Approach 2:
The system performs preliminary data preparation and augmentation before the actual anomaly detection task. By pre-generating synthetic anomalous samples and preparing enhanced training datasets in advance, the system ensures that sufficient learning material is available before deployment, improving detection accuracy without requiring large volumes of actual rare event data
2Reliability
If class balancing methods such as oversampling and undersampling are applied, then the training data distribution can be adjusted, but the system complexity increases and detection effectiveness deteriorates
Solution Approach 1:
The patent extracts and focuses specifically on the rare anomalous events, separating them from the majority normal data. By applying targeted data augmentation only to the rare event classes rather than manipulating the entire dataset, the system avoids the complexity of global class balancing methods while still achieving improved detection accuracy for anomalies
Solution Approach 2:
The system applies different processing strategies to different data classes locally. Instead of uniform oversampling or undersampling across all classes, the patent applies targeted augmentation specifically to rare anomalous events, maintaining simplicity for the majority class while enhancing the rare event detection capability where it is most needed
3Measurement precision
If more training data is collected to improve rare event detection, then detection accuracy can be improved, but the time and resources required for data collection and processing increase
Solution Approach 1:
Instead of collecting additional real-world rare event data which would be time-consuming, the patent creates synthetic copies of anomalous events through data augmentation. This generates sufficient training examples for rare events computationally without requiring extended data collection periods, maintaining detection precision while minimizing time loss
Solution Approach 2:
The system performs data augmentation and synthetic sample generation as a preliminary step that can be completed once before deployment. This preliminary action ensures sufficient training data is available without requiring continuous data collection during operation, reducing the time required for both data collection and model training
Data Source
AI summary
Methods, apparatus, and processor-readable storage media for machine learning-based anomaly detection are provided herein. An example method includes obtaining a set of data record pairs, where each of at least a subset of the data record pairs comprise a target data record to be classified and a historical data record that is labeled as anomalous, and processing the set of data record pairs by at least two neural networks that generate respective embeddings for the data record pairs in the set. The method includes calculating a similarity score between the generated embeddings for each of the data record pairs in the set of data record pairs, classifying the target data record as anomalous in response to determining that at least one of the similarity scores satisfies a first threshold, and initiating one or more automated actions in response to the target data record being classified as anomalous.