Vector-Based Log Anomaly Detection in Cloud Networks
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Current log analytic tools are not scalable to handle the massive and diverse computer performance logs generated by large cloud computing networks, struggling to detect and mitigate complex anomalies in real-time due to their offline operation and lack of optimization for real-time processing.
Innovation Solution
The system determines term characterization values for computer performance logs, generates vectors, and aggregates logs based on similarity scores to identify rare logs, which are then labeled and matched with new logs to detect anomalies, enabling real-time anomaly detection and automated responses.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Productivity
If automated log analytic tools are used to process computer performance logs, then processing speed and real-time detection capability are improved, but the tools cannot handle the massive quantity and diversity of logs effectively
Solution Approach 1:
The patent transforms log data from unstructured text to structured vector representations, changing the parameter space from textual similarity to mathematical vector space. This enables efficient real-time processing while maintaining the ability to handle diverse log formats, as vectors can capture semantic meaning across different log syntaxes and structures.
Solution Approach 2:
The patent replaces traditional text-based log analysis mechanisms with machine learning models that operate in vector space. Instead of manually parsing and comparing log text, the system uses trained models to detect anomalies in vector representations, enabling automated real-time detection of complex anomalies across diverse log sources.
2Measurement precision
If manual analysis of computer performance logs is performed, then detection accuracy for complex anomalies is improved, but scalability and real-time response capability deteriorate
Solution Approach 1:
The patent implements a system where the log analysis performs self-service through automated machine learning models. The models are trained on historical log data and automatically detect anomalies in real-time without manual intervention, maintaining high detection accuracy while achieving scalable automated processing of massive log volumes.
Solution Approach 2:
The patent performs preliminary training of machine learning models on historical log data before deployment. This preliminary action creates pre-trained models that can immediately detect anomalies in real-time logs, combining the accuracy of manual analysis with the speed of automated processing.
3Reliability
If offline log analytic tools are used to detect performance anomalies, then processing completeness is improved, but real-time mitigation capability deteriorates
Solution Approach 1:
The patent transitions from static offline batch processing to dynamic real-time stream processing. The system continuously processes logs as they arrive, maintaining detection completeness through ongoing analysis while enabling immediate real-time response to anomalies, eliminating the time delay inherent in offline tools.
Solution Approach 2:
The patent implements continuous log analysis that operates without interruption. Instead of periodic batch processing, the system continuously analyzes incoming logs in real-time, ensuring no anomalies are missed while enabling immediate detection and response, maintaining both completeness and real-time capability.
Data Source
AI summary
In an exemplary embodiment, computer circuitry determines term characterization values for terms in computer performance logs and generates vectors that indicate the term characterization values. The computer circuitry determines vector similarity scores for these vectors. The computer circuitry aggregates the computer performance logs into aggregated logs based on the vector similarity scores. The computer circuitry selects rare logs from these aggregated logs and obtains computer performance anomaly labels for the rare logs. The computer circuitry matches new computer performance logs with the rare logs to detect the labeled computer performance anomalies.