Log Event Identification via NLP and Inverse Document Frequency
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
In internet-based computing environments, identifying critical log events from the vast number of log entries generated by applications is challenging due to the sheer volume of non-critical, repetitive entries, making it difficult for software engineers to isolate problems and monitor application performance.
Innovation Solution
The use of natural language processing (NLP) and machine learning techniques to filter and rank log events by eliminating non-essential words, comparing similar events, and applying inverse document frequency analysis to identify and prioritize critical log events, thereby reducing the number of entries stored and focusing on key issues.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Loss of information
If all log events are stored and reviewed, then complete information is available, but the volume of data becomes overwhelming and critical events become difficult to identify
Solution Approach 1:
The patent extracts and removes non-essential words (stop words, punctuation, numbers) from log events through NLP processing. This extraction of meaningful content from the bulk of redundant data enables the system to focus on critical information while reducing overall data volume complexity.
Solution Approach 2:
The patent merges similar log events into unified event types through clustering and classification algorithms. By combining repetitive events with identical or semantically equivalent characteristics into single representative events, the system maintains information completeness while dramatically reducing the number of individual entries to manage.
2Productivity
If NLP processing is applied to reduce data volume, then critical events become easier to identify, but processing time and computational resources increase
Solution Approach 1:
The patent performs preliminary NLP processing steps (stop word removal, stemming, lemmatization) on log events before they are stored or analyzed further. This preliminary action prepares the data in advance, reducing the computational burden during subsequent analysis phases and enabling faster critical event identification.
Solution Approach 2:
The patent applies different processing intensities to different parts of the log analysis process. Critical event identification receives intensive NLP processing, while routine event categorization uses lighter processing algorithms. This localized quality approach optimizes processing time by applying computational resources only where most needed.
3Quantity of substance
If similar log events are combined into single entries, then the number of entries is reduced, but the detail and granularity of the log data is lost
Solution Approach 1:
The patent segments log events into hierarchical categories and types, maintaining structural detail while reducing quantity. By organizing events into parent-child category relationships, the system preserves granular information about event characteristics while consolidating repetitive instances into unified entries with representative samples.
Solution Approach 2:
The patent introduces an intermediary representation layer that summarizes similar events while preserving essential characteristics. This intermediary event type representation acts as a mediator between the raw detailed log entries and the consolidated view, maintaining measurement precision through preserved key attributes while reducing the quantity of individual entries.
Data Source
AI summary
Aspects of the disclosure relate to various systems and techniques that provide methods and systems for identifying log event for computing systems. For example, a computing device may receive a log event of an application, and determine a value representative of an inverse relationship between a frequency of the log event and a criticality of the log event. The computing device may initiate an action to address the event indicated by the log event based on a comparison between the determined value and a threshold.