Online Unsupervised Anomaly Detection via hLOF Algorithm
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Current machine learning approaches face challenges in real-time anomaly detection in high-speed data streams due to limited computing resources, particularly in identifying outliers in infinite and unlabeled data streams, which is critical for applications like financial fraud detection where timely and accurate identification is essential.
Innovation Solution
The implementation of an online unsupervised incremental machine learning model, specifically the Streaming Local Outlier Factor based Heterogenous Nearest Neighbors (hLOF) algorithm, which dynamically evaluates local outliers in data streams by using a sliding window and applying classification algorithms like kNN, IkNN, and JkNN to reduce run-time complexity and efficiently process high-dimensional data.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Measurement precision
If batch learning techniques are used to identify anomalous data, then detection accuracy can be improved, but processing speed and real-time capability deteriorate
Solution Approach 1:
The patent segments the batch learning process into online incremental learning steps, where data is processed sequentially in small batches rather than all at once. The sliding window mechanism divides the data stream into manageable segments, allowing the system to maintain detection accuracy while improving processing speed through incremental updates of the anomaly detection model.
Solution Approach 2:
The patent implements dynamic adaptation by continuously updating the anomaly detection model as new data arrives in the stream. The sliding window dynamically adjusts to include recent data while excluding older data, allowing the system to adapt to changing patterns in real-time while maintaining both accuracy and processing efficiency.
2Reliability
If more computing resources are allocated for anomaly detection, then detection capability is improved, but computational complexity and storage requirements increase
Solution Approach 1:
The patent extracts only the essential features and patterns needed for anomaly detection from the data stream, rather than processing the entire dataset. By focusing on local outlier factors and nearest neighbor relationships within a sliding window, the system reduces computational complexity while maintaining detection capability.
Solution Approach 2:
The patent applies local quality by computing anomaly scores based on local neighborhood relationships rather than global data characteristics. The hLOF algorithm calculates local outlier factors for each data point based on its k-nearest neighbors, reducing the computational burden compared to global anomaly detection methods while improving detection of localized anomalies.
3Measurement precision
If traditional outlier analysis is applied to infinite data streams, then comprehensive anomaly detection is achieved, but storage space and run-time complexity increase
Solution Approach 1:
The patent performs preliminary actions by pre-computing and storing only the essential statistics and nearest neighbor relationships within the sliding window, rather than retaining all historical data. This allows the system to quickly compute anomaly scores for new data points without reprocessing the entire data stream, significantly reducing run-time complexity.
Solution Approach 2:
The patent implements periodic action through the sliding window mechanism, which periodically updates the dataset by adding new data points and removing old ones. This periodic refresh allows the system to maintain comprehensive outlier detection coverage while keeping storage requirements and computational complexity bounded by the window size rather than the infinite data stream.
Data Source
AI summary
A computerized-method for real-time detection of anomalous data, by processing high-speed streaming data. In a computerized-system receiving a data-stream comprised of unlabeled data points, and operating an Anomalous Data Detection (ADD) module. The ADD module receives at least one of: (i) k number of data point neighbors for each data point; (ii) X number of data points in a predetermined period of time; (iii) d number of dimensions of each data point, threshold; and (iv) n number of data points that said ADD module is operating on, in a predefined time unit. Then, the ADD module prepares a dataset having n data points from the received X data points; and then identifies one or more data points, from the received data stream, as outliers to send an alert with details related to the identified outliers, thus, dynamically evaluating local outliers in the received data stream.