End-User Session Generation from System Log Clustering

Resolve Bottlenecks,
Find Innovative SolutionsGenerate Solutions

Solution Overview

Problem

In collaborative research and development environments, it is challenging to differentiate between end-user activities and system-generated activities from the multitude of heterogeneous system log files, leading to inaccurate information mining and labor-intensive data processing.

Innovation Solution

A method that collects system log files, clusters messages using a clustering algorithm to generate message templates, identifies potential end-user events by correlating them with known user-initiated events based on temporal correlations, and distinguishes between end-user and system events using a seed set of known event types.

Engineering Contradictions & Design Principles

VSEngineering Contradiction Analysis

1Measurement precision

If system log files are mined to identify end-user activities, then information mining accuracy can be improved, but the complexity of differentiating between end-user and system events increases

Engineering Contradiction:

Improveinformation mining accuracyVSAvoidevent differentiation complexity

Core Design Contradiction:

Measurement precisionVSDevice complexity

Solution Approach 1:

The patent segments log messages into distinct categories: end-user events, system events, and ambiguous events. This segmentation allows the system to process different types of events differently, improving accuracy by focusing analysis on end-user events while filtering out system-generated events through the clustering and template generation process.

Inventive Principle:

Principle #1Segmentation

Solution Approach 2:

The patent introduces message templates as an intermediary representation between raw log messages and event classification. These templates serve as mediators that capture the essential patterns of end-user events, enabling accurate differentiation without directly analyzing the complexity of individual log messages.

Inventive Principle:

Principle #24Intermediary (Mediator)

2Measurement precision

If manual analysis of system logs is performed to identify end-user events, then event identification accuracy improves, but processing time and labor increase

Engineering Contradiction:

Improveevent identification accuracyVSAvoiddata processing time

Core Design Contradiction:

Measurement precisionVSLoss of time

Solution Approach 1:

The system performs self-service by automatically generating message templates and classifying events without requiring manual intervention. The clustering algorithm automatically identifies patterns in log messages, and the template generation process automatically creates representations of end-user events, eliminating the need for manual analysis while maintaining high accuracy.

Inventive Principle:

Principle #25Self-service

Solution Approach 2:

The patent replaces manual mechanical analysis with automated computational processes. Instead of human analysts manually reviewing logs, the system uses clustering algorithms and template matching to automatically identify and classify end-user events, significantly reducing processing time while maintaining or improving accuracy through computational pattern recognition.

Inventive Principle:

Principle #28Mechanics substitution (Replace mechanical system)

3Quantity of substance

If all log messages are processed to identify end-user activities, then completeness of event data improves, but system resource consumption increases

Engineering Contradiction:

Improveevent data completenessVSAvoidsystem resource consumption

Core Design Contradiction:

Quantity of substanceVSUse of energy by moving object

Solution Approach 1:

The patent extracts only the relevant portion of log data by identifying and isolating end-user events from system events. Through clustering and template generation, the system extracts meaningful patterns from the log messages and discards redundant system-generated events, thereby reducing the quantity of data that requires further processing and lowering system resource consumption.

Inventive Principle:

Principle #2Taking out (Extraction)

Solution Approach 2:

The system applies partial action by processing only the subset of log messages that are likely to represent end-user events, rather than processing every log message in detail. The clustering algorithm preliminarily sorts messages into relevant groups, allowing the system to apply more intensive analysis only to the necessary portion of data, thus balancing completeness with resource efficiency.

Inventive Principle:

Principle #16Partial or excessive action

Data Source

PatentUS11604715B2Generation of end-user sessions from end-user events identified from computer system logs

Publication Date: 2023.03.14 INTERNATIONAL BUSINESS MACHINE CORPORATION

AI summary

One embodiment provides a method including: collecting system log files comprising a plurality of log messages representing activity within the system; generating a plurality of message templates by (i) clustering the plurality of messages into groups having similar activity patterns and (ii) generating a message template for each of the groups; identifying message templates of interest and determining the message templates representing actual end-user events as opposed to system events, wherein the determining comprises (i) identifying a seed set of event types that represent known end-user events based upon identifying the event as being known to be initiated by a user and (ii) correlating the event to the known end-user event based upon a time correlation between the known end-user event and the event within the message template of interest.