Anomaly Detection via Hash Signature Segmentation
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Current machine learning models for anomaly detection in IT systems are inefficient and costly due to the need for individual training and scoring of each object, especially when not all objects are present, leading to resource-intensive processing and prolonged training times.
Innovation Solution
The use of multiple hashing algorithms to generate unique hash signatures for monitored objects, aggregating metric values within these signatures, and training machine learning models on these aggregated values, allowing for efficient identification of anomalous objects through iterative filtering across multiple hash levels.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Measurement precision
If machine learning models are trained and scored for each object individually, then anomaly detection accuracy is improved, but resource consumption and training time increase significantly
Solution Approach 1:
The patent segments the set of all possible objects into multiple subsets, where each subset contains objects that share the same hash signature value. This segmentation allows the system to train and score models on aggregated data within each subset rather than treating every object individually, thereby reducing computational overhead while maintaining detection accuracy through the use of multiple hashing algorithms
Solution Approach 2:
The patent combines multiple hashing algorithms to generate hash signatures that group objects into subsets. By merging the results from multiple hashing algorithms, the system can identify anomalous objects through intersection of anomalous subsets, achieving accurate anomaly detection with reduced training requirements compared to individual object processing
2Measurement precision
If machine learning models are trained for each object individually, then detection accuracy is improved, but the time required for training and implementation becomes unacceptable
Solution Approach 1:
The patent performs preliminary actions by pre-computing hash signatures for all possible objects and organizing them into subsets before actual anomaly detection is needed. This preliminary organization enables rapid scoring during operation, as the system only needs to evaluate aggregated metrics for existing subsets rather than training models for each object in real-time
Solution Approach 2:
By segmenting objects into hash-based subsets, the system reduces the number of model training operations required. Instead of training one model per object, the system trains models on aggregated data from multiple objects within each subset, dramatically reducing training time while maintaining detection capability
3Measurement precision
If conventional techniques are used to score values across a sequence of objects, then individual object accuracy is maintained, but resource consumption becomes significant
Solution Approach 1:
The patent merges multiple hashing algorithms to create a comprehensive subset structure. By combining the subset structures from multiple hashing algorithms and finding intersections of anomalous subsets, the system achieves accurate identification of anomalous objects while significantly reducing the computational resources required compared to processing each object separately
Data Source
AI summary
Described systems and techniques provide fast, efficient, and cost-effective techniques for detecting anomalous behaviors of monitored objects. Multiple hashing algorithms, each providing multiple hash bins, may be used to generate a unique hash signature for each of the monitored objects. Metric values characterizing the behavior of the monitored objects may be aggregated within individual ones of the multiple hash bins of each of the multiple hashing algorithms. Then, one or more machine learning models may be trained using the unique hash signatures and their included, aggregated metric values. During subsequent scoring using the trained machine learning model(s), each of the aggregated metric values of each of the hash bins may be scored, and a single or small subset of anomalous objects may be identified.