Anomaly Detection via Log Message Segmentation

Resolve Bottlenecks,
Find Innovative SolutionsGenerate Solutions

Solution Overview

Problem

Existing anomaly detection systems in server and network systems are unable to detect unknown anomalies, as they rely on blacklist monitoring and manual keyword searches in text logs.

Innovation Solution

An anomaly detection apparatus and method that uses a storage unit to associate partial character strings of messages with IDs, a classification unit to classify messages by type, and a detection unit to identify anomalies based on assigned IDs, enabling the detection of unknown anomalies by monitoring changes in the number of new IDs assigned over time.

Engineering Contradictions & Design Principles

VSEngineering Contradiction Analysis

1Reliability

If manual keyword search and blacklist monitoring are used for anomaly detection, then known anomalies can be detected, but unknown anomalies cannot be detected

Engineering Contradiction:

Improveanomaly detection capabilityVSAvoiddetection of unknown anomalies

Core Design Contradiction:

ReliabilityVSAdaptability or versatility

Solution Approach 1:

The patent segments log messages into fixed-length character strings (e.g., 3-character segments) and creates multiple segmentations of the same message. This segmentation allows the system to detect anomalies by comparing segment frequencies without requiring pre-defined keywords or blacklists, thereby enabling detection of unknown anomaly types while maintaining reliability for known anomalies.

Inventive Principle:

Principle #1Segmentation

Solution Approach 2:

The patent replaces the manual keyword search mechanism with an automated statistical analysis system that uses character string segmentation and frequency counting. This substitution eliminates the need for manual configuration of detection rules while improving both the detection of known anomalies and the ability to discover unknown anomaly patterns through automated analysis of message characteristics.

Inventive Principle:

Principle #28Mechanics substitution (Replace mechanical system)

2Loss of information

If manual keyword search is used in text logs, then specific critical messages can be extracted, but the process is time-consuming and cannot detect unknown anomalies

Engineering Contradiction:

Improvecritical message extractionVSAvoidmanual search time

Core Design Contradiction:

Loss of informationVSLoss of time

Solution Approach 1:

The system performs self-service by automatically analyzing log messages through character string segmentation and frequency statistics without requiring manual intervention. The anomaly detection apparatus autonomously processes text logs, extracts critical information, and identifies anomalies based on segmented character patterns, eliminating the time-consuming manual search process while maintaining comprehensive information extraction.

Inventive Principle:

Principle #25Self-service

Solution Approach 2:

The patent substitutes manual keyword searching with an automated computational system that segments messages into character strings and analyzes frequency distributions. This mechanical substitution transforms the time-intensive manual process into rapid automated processing, significantly reducing time loss while improving the ability to detect both known and unknown anomalies through systematic analysis.

Inventive Principle:

Principle #28Mechanics substitution (Replace mechanical system)

3Ease of manufacture

If blacklist monitoring is used for anomaly detection, then detection rules are simple to implement, but only pre-defined anomalies can be detected

Engineering Contradiction:

Improvedetection system implementationVSAvoidanomaly type coverage

Core Design Contradiction:

Ease of manufactureVSAdaptability or versatility

Solution Approach 1:

The patent implements a universal detection mechanism that uses character string segmentation applicable to any log message format. By segmenting messages into standardized character units and analyzing frequency patterns, the system achieves multi-functionality that can detect various anomaly types (known and unknown) without requiring separate detection rules for each anomaly category, thus maintaining ease of implementation while expanding anomaly type coverage.

Inventive Principle:

Principle #6Universality (Multi-functionality)

Solution Approach 2:

The system changes the detection parameter from keyword-based matching to character string frequency analysis. By transforming the detection approach to use segmented character strings and their frequency statistics, the system maintains simple implementation through standardized processing while gaining the ability to detect diverse anomaly types through parameter-based analysis rather than rule-based matching.

Inventive Principle:

Principle #35Parameter changes

Data Source

PatentUS11757708B2Anomaly detection device, anomaly detection method, and anomaly detection program

Publication Date: 2023.09.12 NIPPON TELEGRAPH & TELEPHONE CORP

AI summary

An anomaly detection apparatus (10) includes a storage unit (14) that stores dictionary information (14b) in which a partial character string of a message representing a type of a message included in a text log output from a system and an ID set for the type of the message are associated with each other. The anomaly detection apparatus (10), when the message included in the text log output from the system is acquired, refers to the dictionary information (14b) stored in the storage unit (14), classifies the message included in the text log by the type, and assigns the ID to the message that has been classified; and detects an anomaly based on the ID assigned to the message.