Graph-Based Log Sequence Anomaly Detection in Mainframes

Resolve Bottlenecks,
Find Innovative SolutionsGenerate Solutions

Solution Overview

Problem

Existing methods for analyzing mainframe logs are inefficient due to the complexity of interpreting highly technical information, and they lack context for effective diagnosis, with limited performance in log sequence clustering and diagnosis.

Innovation Solution

The implementation of graph-based anomaly detection and problem diagnosis using machine learning and natural language processing techniques, which involves parsing and serializing log sequences, generating vector representations, and building directional graphs to identify anomalies by comparing new log sequences to historical sequences.

Engineering Contradictions & Design Principles

VSEngineering Contradiction Analysis

1Productivity

If traditional methods are used to analyze mainframe logs, then the analysis process becomes manageable, but the efficiency and diagnostic capability are insufficient due to the complexity of interpreting highly technical information

Engineering Contradiction:

Improvelog analysis efficiencyVSAvoidcomplexity of interpreting technical information

Core Design Contradiction:

ProductivityVSDevice complexity

Solution Approach 1:

The patent introduces an intermediary system that includes a graph generator and anomaly detector. This intermediary translates complex mainframe log sequences into graph representations, making the technical information more manageable and interpretable while maintaining high analysis efficiency

Inventive Principle:

Principle #24Intermediary (Mediator)

Solution Approach 2:

The patent replaces traditional manual or rule-based log analysis mechanisms with machine learning-based anomaly detection. The system automatically learns patterns from historical log sequences and detects anomalies without requiring manual interpretation of complex technical information

Inventive Principle:

Principle #28Mechanics substitution (Replace mechanical system)

2Measurement precision

If detailed analysis of log sequences is performed to improve diagnosis quality, then diagnostic accuracy improves, but the time and computational resources required increase

Engineering Contradiction:

Improvediagnosis accuracyVSAvoidanalysis time

Core Design Contradiction:

Measurement precisionVSLoss of time

Solution Approach 1:

The patent performs preliminary actions by pre-processing historical log sequences into graph representations and storing them in a database. When analyzing new logs, the system quickly compares against pre-processed historical data, achieving high diagnostic accuracy without excessive analysis time

Inventive Principle:

Principle #10Preliminary action

Solution Approach 2:

The patent segments the log analysis process into distinct components: log sequence generation, graph construction, database storage, and anomaly detection. This segmentation allows each component to be optimized independently, balancing diagnostic precision with analysis speed

Inventive Principle:

Principle #1Segmentation

Data Source

PatentUS11474892B2Graph-based log sequence anomaly detection and problem diagnosis

Publication Date: 2022.10.18 INTERNATIONAL BUSINESS MACHINE CORPORATION

AI summary

Techniques include generating a log sequence for new logs that have been received, searching a log sequence database for the log sequence having been generated, and determining that the log sequence is anomalous in response to not finding an identical log sequence in the log sequence database. In response to the log sequence not being found in the log sequence database, the log sequence is compared to a graph of historical log sequences to find a closest sequence path to one or more historical log sequences. An anomaly of the log sequence is diagnosed based on an occurrence at which the log sequence deviates from the closest sequence path of the one or more historical log sequences.