Automated Anomaly Detection in Heterogeneous Log Streams

Resolve Bottlenecks,
Find Innovative SolutionsGenerate Solutions

Solution Overview

Problem

Manual inspection of logs in modern systems is arduous and time-consuming, especially in large-scale connected components and devices, as conventional systems rely on human understanding and keyword searches, making it difficult to quickly identify the root cause of errors.

Innovation Solution

A log analytics platform (NGLA) uses machine learning to generate regular expression patterns from heterogeneous log sources, tokenizes logs, and stores models in a global database for anomaly detection, enabling automated anomaly alerts without human input, and allowing for real-time monitoring and visualization of potential causes.

Engineering Contradictions & Design Principles

VSEngineering Contradiction Analysis

1Measurement precision

If manual inspection of logs is used, then human understanding and keyword search capability are maintained, but the time and effort required to identify root cause increases significantly

Engineering Contradiction:

Improveroot cause identification accuracyVSAvoiddebugging time

Core Design Contradiction:

Measurement precisionVSLoss of time

Solution Approach 1:

The patent replaces manual mechanical inspection of logs with automated machine learning-based anomaly detection. The system uses unsupervised learning algorithms to automatically analyze log streams, generate anomaly scores, and identify root causes without human intervention, thereby reducing debugging time while maintaining or improving detection accuracy.

Inventive Principle:

Principle #28Mechanics substitution (Replace mechanical system)

Solution Approach 2:

The patent introduces an intermediary anomaly detection service that sits between the log sources and human operators. This service processes logs through multiple analysis stages (pattern recognition, anomaly scoring, root cause identification) and presents processed insights to users, filtering out noise and highlighting only critical anomalies that require human attention.

Inventive Principle:

Principle #24Intermediary (Mediator)

2Adaptability or versatility

If conventional log search systems are used, then simple keyword search capability is provided, but the ability to detect anomalies in heterogeneous log sources is limited

Engineering Contradiction:

Improveheterogeneous log source compatibilityVSAvoidsystem complexity

Core Design Contradiction:

Adaptability or versatilityVSDevice complexity

Solution Approach 1:

The patent creates a universal anomaly detection service that can process multiple types of log sources (system logs, application logs, sensor data) through a single unified platform. The system uses format-agnostic pattern recognition and normalization techniques to handle heterogeneous log structures, providing versatile anomaly detection across diverse data sources without requiring separate specialized tools for each log type.

Inventive Principle:

Principle #6Universality (Multi-functionality)

Data Source

PatentUS9928155B2Automated anomaly detection service on heterogeneous log streams

Publication Date: 2018.03.27 NEC CORP

AI summary

Systems and methods are disclosed for handling log data from one or more applications, sensors or instruments by receiving heterogeneous logs from arbitrary/unknown systems or applications; generating regular expression patterns from the heterogeneous log sources using machine learning and extracting a log pattern therefrom; generating models and profiles from training logs based on different conditions and updating a global model database storing all models generated over time; tokenizing raw log messages from one or more applications, sensors or instruments running a production system; transforming incoming tokenized streams are into data-objects for anomaly detection and forwarding of log messages to various anomaly detectors; and generating an anomaly alert from the one or more applications, sensors or instruments running a production system.