Real-time Log Event Categorization via TF-IDF Vector Clustering
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
The manual analysis and categorization of log events in computing systems is time-consuming, prone to errors, and not feasible due to the high volume, high dimensionality, and variability of log data, especially in real-time environments where thousands of log categories with different formats are generated continuously, making it difficult for operators to detect abnormalities and perform efficient monitoring.
Innovation Solution
A system and method that utilize Term Frequency-Inverse Document Frequency (TF-IDF) vectors and centroid matrices to categorize real-time log events by calculating distances and adapting centroid matrices based on new words, allowing for real-time categorization into pre-existing or new log categories using a silhouette threshold, thereby reducing manual effort and error.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If manual analysis and categorization of log events is performed, then operators can detect abnormalities and monitor computing systems, but the process becomes time-consuming and not feasible for high volume log data
Solution Approach 1:
The patent replaces the mechanical manual analysis system with an automated computational system using TF-IDF vectors and centroid-based categorization. The system automatically computes similarity scores between log events and categories, eliminating the need for manual operator intervention while maintaining or improving detection accuracy.
Solution Approach 2:
The categorization system performs self-service by automatically processing log events through predefined algorithms. The system independently computes TF-IDF vectors, calculates similarities to category centroids, and assigns categories without requiring external manual input, enabling high-volume processing at scale.
2Adaptability or versatility
If operators manually categorize thousands of log categories with different formats, then comprehensive monitoring is achieved, but the complexity and error-proneness increase significantly
Solution Approach 1:
The patent transforms the complexity of handling diverse log formats into a standardized parameter-based system. By converting all log events into TF-IDF vectors regardless of their original format, the system uniformly represents heterogeneous data, simplifying the categorization process while maintaining adaptability to various log structures.
Solution Approach 2:
The TF-IDF vector serves as an intermediary representation between raw log events in various formats and the category classification system. This intermediate vector representation standardizes diverse input formats, enabling consistent processing without requiring complex format-specific handling logic.
3Productivity
If real-time categorization is implemented using automated methods, then processing speed increases, but accuracy may deteriorate without proper algorithms
Solution Approach 1:
The system performs preliminary action by pre-computing TF-IDF vectors for category centroids and storing them for rapid comparison. This pre-processing enables real-time categorization to proceed efficiently by only requiring computation of the log event vector and similarity calculation, rather than full-text analysis during real-time processing.
Solution Approach 2:
The patent replaces manual categorization mechanics with automated computational mechanics using TF-IDF similarity scoring. The system systematically computes cosine similarity between log vectors and category centroids, providing consistent and accurate real-time categorization that scales with processing throughput.
Data Source
AI summary
Embodiments for categorizing a real-time log event are described. In one example, a Term Frequency-Inverse Document Frequency (TF-IDF) vector for the log event is computed based on pre-calculated TF-IDF matrix of log corpus and number of new words in log event, where log corpus comprises one or more pre-existing log events, and where the log event is indicative of error message. Further, distance between TF-IDF vector and cluster centroid of each cluster in the log corpus is calculated. Thereafter, cluster having closest cluster centroid is identified from amongst the clusters based on distance between TF-IDF vector and cluster centroid of each of the clusters, where closest cluster centroid is cluster centroid closest to TF-IDF vector. Subsequently, log event is categorized into one or more log categories based on comparison of distance between TF-IDF vector and closest cluster centroid pre-determined silhouette threshold corresponding to cluster with closest cluster centroid.