Scalable Anomaly Detection via Weighted Anchor Aggregation
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Current anomaly detection systems in distributed systems, particularly those involving third-party updates, face challenges with incorrect comparison values, leading to false positives and negatives, and are unable to handle newly added database updates effectively, resulting in data quality and control issues.
Innovation Solution
A scalable anomaly detection framework that utilizes an ensemble of trained models to detect anomalous values by generating optimal anchor values through weighted aggregation of non-anomalous source-specific anchor values, enabling real-time detection and handling of decentralized network environments with limited or low-quality data.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If basic logic is used to select comparison values for anomaly detection, then the system is simple to implement, but it produces false positive and false negative identifications and cannot provide up-to-date values
Solution Approach 1:
The system segments the anomaly detection task by creating separate classification models for different data sources (first-party and third-party updates). Each model is trained independently on source-specific anchor values, allowing specialized detection logic for each data source type while maintaining overall system manageability.
Solution Approach 2:
The system performs preliminary training of classification models using historical anchor values before actual anomaly detection occurs. The models are pre-trained on labeled data to recognize patterns, and anchor values are pre-computed and stored for comparison during real-time anomaly detection operations.
2Adaptability or versatility
If basic comparison logic is used, then the system is easy to operate, but it cannot handle newly added database updates and provides outdated comparison values
Solution Approach 1:
The system dynamically adapts to new database update types by training new classification models for each source when needed. The architecture allows flexible addition of new models without requiring changes to existing operational logic, enabling the system to handle evolving data structures while maintaining straightforward operation through standardized model interfaces.
Solution Approach 2:
The system incorporates feedback mechanisms where anchor values are continuously updated based on observed data patterns. Classification models receive feedback from actual anomaly detection results and can be retrained with new information, allowing the comparison values to remain current and adaptive to changing data characteristics.
3Adaptability or versatility
If third party updates are incorporated, then the system handles decentralized data sources, but data quality and control concerns arise with unreliable participants
Solution Approach 1:
The system applies local quality control by creating source-specific classification models that are trained on anchor values from each particular data source. Each model learns the characteristic patterns and quality expectations of its specific source, enabling tailored anomaly detection that accounts for the unique reliability characteristics of different decentralized participants.
Solution Approach 2:
The classification models act as intermediaries between the decentralized third-party update sources and the central anomaly detection system. These models filter and validate incoming data by comparing it against source-specific anchor values, mediating the trust relationship and preventing unreliable data from affecting system operations.
Data Source
AI summary
Systems and methods of anomaly detection using an optimal reference value are disclosed. A plurality of source-specific anchor values are received and a plurality of model features are generated. A plurality of trained source-specific classification models, each associated with at least one of the plurality of source-specific anchor values and each configured to receive a subset of the plurality of model features, are implemented. Each of the plurality of trained source-specific classification models is configured to classify the associated at least one of the plurality of source-specific anchor values as one of anomalous or non-anomalous. A trained weighted classification model is implemented to generate an optimal anchor value. The optimal anchor value includes a weighted aggregation of each of the plurality of source-specific anchor values identified as non-anomalous. An optimal reference value is generated based on the optimal anchor value.