Edge Log Anomaly Detection Using Dominant Pattern Extraction
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing anomaly detection systems face challenges in efficiently processing and transporting large volumes of log data due to bandwidth and computational limitations, hindering effective anomaly analysis.
Innovation Solution
An anomaly service identifies dominant patterns in log data from edge nodes, extracts non-anomalous blocks, generates anomaly vectors, and distributes them to nodes for efficient detection of anomalous behavior by comparing sequence vectors with anomaly vectors using cosine similarity.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Measurement precision
If all log data is transported to centralized server for pattern extraction, then anomaly detection accuracy is improved, but bandwidth consumption increases significantly
Solution Approach 1:
The patent extracts only the essential anomaly-related information from log data at the edge nodes, creating compact anomaly vectors that capture abnormal patterns without transmitting complete log data. This extraction approach maintains detection accuracy by preserving anomaly characteristics while dramatically reducing the volume of data that needs to be transported to centralized servers.
Solution Approach 2:
The system segments the anomaly detection process into distributed edge computation and centralized coordination. Edge nodes independently extract anomaly vectors from local log data, then only these extracted anomaly indicators are transmitted to the centralized server for aggregation and final detection decisions, reducing overall bandwidth requirements.
2Measurement precision
If complete log data is processed for pattern extraction, then detection accuracy is improved, but computational overhead increases
Solution Approach 1:
The patent applies extraction by identifying and isolating only the anomaly-relevant features from log data at the edge nodes. Instead of processing complete log entries, the system extracts compact anomaly vectors that preserve essential abnormal patterns, significantly reducing computational requirements while maintaining detection accuracy.
Solution Approach 2:
The system performs preliminary anomaly vector extraction at edge nodes before data leaves the local environment. This preliminary processing filters out normal operational patterns and pre-packages only suspicious anomalies for further centralized analysis, reducing the computational burden on both edge and centralized systems.
3Measurement precision
If anomaly detection is performed at centralized server, then comprehensive analysis is achieved, but response time increases
Solution Approach 1:
The patent implements preliminary anomaly vector extraction at edge nodes, preparing anomaly indicators locally before transmission. This preliminary action enables faster response times because the system doesn't need to wait for complete log data to be collected and processed centrally - anomaly vectors are generated and transmitted immediately when anomalies are detected at the edge.
Solution Approach 2:
The detection process is segmented into fast edge-based anomaly vector generation and slower centralized aggregation. This segmentation allows the time-critical extraction phase to occur locally with minimal latency, while the less time-sensitive comprehensive analysis can be performed centrally without compromising overall response time.
Data Source
Figure 1
Figure 2
Figure 3
AI summary
An anomaly service receives log data from nodes in a computing environment, which includes a sequence of information indicative of log messages produced by the nodes. The anomaly service identifies dominant patterns in the sequence of information that are representative of non-anomalous blocks of the log messages. Having identified the dominant patterns, the service is able to extract the non-anomalous blocks from the log data to reveal anomalous blocks that do not fit the dominant patterns. The service may then generate anomaly vectors based on the anomalous blocks, which can be distributed to the nodes to detect anomalies.