Log Analysis Device Using Nonnegative Matrix Factorization
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Conventional system monitoring using text logs is inefficient in analyzing massive amounts of data, leading to overlooked useful information due to the complexity of system configurations and the manual effort required for keyword-based searches.
Innovation Solution
An analysis device and method that classify text logs into templates, create a log matrix, perform nonnegative matrix factorization to extract patterns, remove frequent patterns, and determine the importance of extracted patterns to identify critical messages, facilitating efficient anomaly detection and information extraction.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Productivity
If manual keyword-based search is used to extract critical messages from text logs, then the analysis process is simple and easy to understand, but it becomes inefficient and time-consuming when dealing with massive amounts of log data
Solution Approach 1:
The patent replaces manual keyword-based search (mechanical human operation) with automated machine learning algorithms including non-negative matrix factorization (NMF), clustering algorithms, and natural language processing. The analysis device automatically extracts patterns, identifies anomalies, and classifies log messages without human intervention, dramatically improving productivity while reducing time loss.
Solution Approach 2:
The system enables self-service log analysis by automatically performing pattern extraction, anomaly detection, and message classification without requiring manual keyword specification. The NMF algorithm autonomously identifies log patterns and their relationships, allowing the system to serve its own analysis needs without external human input.
2Loss of information
If keyword-based extraction is used to identify critical messages, then the method is straightforward, but useful information in non-matched messages may be overlooked
Solution Approach 1:
The patent implements a universal analysis framework that performs multiple functions simultaneously: pattern extraction, clustering, anomaly detection, and information extraction. Instead of relying on a single keyword-matching function, the system uses NMF to extract multiple log patterns and their relationships, ensuring comprehensive coverage of useful information across all log messages regardless of specific keywords.
Solution Approach 2:
The system transforms the analysis approach by changing parameters from fixed keywords to dynamic patterns extracted through NMF. The clustering algorithm adjusts cluster assignments based on log content similarities, and anomaly detection thresholds are dynamically determined, allowing the system to adapt to different log types and identify useful information that would be missed by static keyword matching.
3Adaptability or versatility
If templates are determined to extract event candidates from log messages, then event clustering becomes possible, but the template determination process adds system complexity
Solution Approach 1:
The patent replaces manual template creation with automated template determination using non-negative matrix factorization. The NMF algorithm automatically discovers log patterns and generates templates from raw log data without human intervention. This substitution maintains the benefit of event clustering while eliminating the complexity of manual template design and maintenance.
Data Source
Figure 1
Figure 2
Figure 3
AI summary
A classification unit (151) classifies messages included in a text log depending on types, and gives an ID set for each type to each of the classified messages. A creation unit (152) creates, based on dates of occurrence attached to the messages, a matrix indicating an appearance distribution of the messages in the text log for each predetermined duration for each ID. A pattern extraction unit (153) extracts a plurality of patterns, which are combinations of the IDs, from the matrix created by the creation unit (152). A removal unit (154) removes a part or whole of the patterns from the matrix. A determination unit (155) calculates a degree of importance for each element included in each of the patterns, and determines whether the degree of importance is equal to or higher than a predetermined threshold. A sequence extraction unit (157) extracts a particular sequence from sequences of IDs determined by the determination unit (155) to have a degree of importance equal to or higher than a predetermined threshold.