Anomaly Detection Model Using Latent Hypersphere Thresholds
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing anomaly detection systems in the ISR field face challenges due to the scarcity of previous threat examples, evolving nature of anomalies, and the difficulty in distinguishing between known and unknown threats, leading to noisy and low-precision detections.
Innovation Solution
A computer-implemented method using outlier exposure to train an anomaly detection model by encoding data within a hypersphere, defining a threshold to distinguish normal and anomalous behavior, and employing a neural network for mapping and classification.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Adaptability or versatility
If unsupervised anomaly detection is used to identify unknown unknowns, then the ability to detect new threat types is improved, but precision deteriorates due to low discrimination capability
Solution Approach 1:
The system performs preliminary encoding of both normal and anomalous data into a shared latent space before anomaly detection. By pre-encoding anomalous patterns into the model during training, the system creates a reference framework that enables subsequent detection of unknown anomaly types while maintaining precision through the structured latent space organization
Solution Approach 2:
The patent transforms data from original feature space into a compressed latent representation space through neural network encoding. This dimensional transformation enables the model to capture essential patterns while filtering noise, allowing detection of novel anomalies in the latent space without sacrificing precision in the original data domain
2Measurement precision
If supervised learning with large volumes of labelled anomalous data is used, then precision is improved, but the requirement for large training datasets worsens the practical applicability
Solution Approach 1:
The system extracts and encodes only the essential anomalous patterns into a compact latent representation during training. By extracting the core characteristics of anomalies into the latent space rather than storing full labelled datasets, the system achieves high precision with minimal training data requirements
Solution Approach 2:
The patent changes the representation parameters of anomalous data by encoding them through a neural network into a compressed latent form. This parameter transformation allows the model to learn from small datasets by capturing essential anomaly characteristics in a condensed representation that generalizes well to unseen cases
3Measurement precision
If rules-based systems are used to detect known threats, then precision is improved, but the ability to detect new threat types deteriorates
Solution Approach 1:
The system performs preliminary encoding of anomalous patterns during training, creating a latent space representation of known threats. This preliminary action enables the model to recognize established threat patterns with high precision while the learned latent representations naturally generalize to detect novel threat types that share similar underlying patterns
Solution Approach 2:
The neural network encoder serves multiple functions: it compresses data for efficient storage, learns robust representations for precision detection, and generalizes to detect unknown anomaly types. This universal encoding approach replaces multiple specialized rules with a single adaptable system
Data Source
Figure 1
Figure 2
Figure 3
AI summary
There is provided a computer-implemented method of training an anomaly detection model for detecting anomalies in data associated with the behaviour of land, sea, air and/or space domain assets, comprising: selecting a training dataset for training the anomaly detection model by outlier exposure, the training dataset comprising data associated with the behaviour of assets in at least one of land, sea, air and/or space domain, the training dataset comprising normal data associated with normal behaviour and anomalous data associated with anomalous behaviour; encoding the training dataset within the anomaly detection model to create an encoded training dataset comprising encoded normal data and encoded anomalous data; determining, based on the encoded training dataset, a hypersphere defining a threshold to distinguish the encoded normal data from the encoded anomalous data.