Spatio-Temporal Log Clustering for Distributed Incident Explanation

Resolve Bottlenecks,
Find Innovative SolutionsGenerate Solutions

Solution Overview

Problem

Debugging distributed cloud applications is challenging due to their complexity, with multiple components spread across a distributed infrastructure, making it difficult to identify the cause of outages and incidents effectively.

Innovation Solution

The use of spatio-temporal log clustering to detect anomalies and generate explanations by analyzing log entries, combining spatial information from service deployment topology and temporal information from streaming logs, allowing for the identification of persistent anomalies and composition of incident explanations.

Engineering Contradictions & Design Principles

VSEngineering Contradiction Analysis

1Adaptability or versatility

If distributed cloud applications use fine-grained service-oriented architecture to improve flexibility and scalability, then the system can respond rapidly to demand, but debugging and incident identification become significantly more difficult due to the complexity of multiple components across distributed infrastructure

Engineering Contradiction:

Improveflexibility and scalabilityVSAvoiddifficulty of identifying incident causes

Core Design Contradiction:

Adaptability or versatilityVSDifficulty of detecting and measuring

Solution Approach 1:

The patent segments the complex distributed system into individual service components, each with its own log entries. By analyzing logs at the service level rather than treating the entire distributed system as a monolith, the system can isolate and identify incidents within specific services, making debugging manageable despite the overall system complexity.

Inventive Principle:

Principle #1Segmentation

Solution Approach 2:

The patent introduces log entries as an intermediary layer between the distributed services and the analysis system. These logs serve as mediators that capture and transmit information about service behavior, anomalies, and incidents, enabling indirect observation and analysis of the complex distributed system without requiring direct access to all service components simultaneously.

Inventive Principle:

Principle #24Intermediary (Mediator)

2Reliability

If multiple log entries are analyzed to identify incidents in distributed systems, then comprehensive coverage is achieved, but the time required to process and analyze the large volume of log data increases significantly

Engineering Contradiction:

Improvecomprehensive incident detectionVSAvoidtime to process log data

Core Design Contradiction:

ReliabilityVSLoss of time

Solution Approach 1:

The patent extracts only the relevant and anomalous log entries from the vast volume of distributed system logs. By using anomaly detection to identify unusual patterns and focusing analysis only on those specific entries rather than processing all logs uniformly, the system achieves comprehensive incident detection while significantly reducing the time required for log analysis.

Inventive Principle:

Principle #2Taking out (Extraction)

Solution Approach 2:

The patent applies partial action by analyzing only the subset of log entries that contain anomalies or incident indicators, rather than performing exhaustive analysis on every log entry. This selective approach maintains reliable incident detection by focusing on critical data while reducing overall processing time through avoiding unnecessary analysis of normal operations.

Inventive Principle:

Principle #16Partial or excessive action

3Ease of operation

If traditional log analysis methods are used without spatial and temporal context, then simple processing is maintained, but the ability to distinguish persistent anomalies from transient issues is lost

Engineering Contradiction:

Improvesimplicity of log processingVSAvoidability to identify persistent anomalies

Core Design Contradiction:

Ease of operationVSMeasurement precision

Solution Approach 1:

The patent adds temporal and spatial dimensions to traditional log analysis. By incorporating time-based information (when anomalies occur, duration, frequency) and service topology information (which services are affected, their relationships), the system transforms simple log entry analysis into multi-dimensional pattern recognition, enabling distinction between persistent and transient anomalies while maintaining operational simplicity through automated processing.

Inventive Principle:

Principle #17Another dimension (Dimensionality change)

Data Source

PatentUS12050506B2Generating incident explanations using spatio-temporal log clustering

Publication Date: 2024.07.30 INTERNATIONAL BUSINESS MACHINE CORPORATION

AI summary

An embodiment includes detecting a set of anomalies recorded during a first predefined window of time in log entries for a computer environment. The embodiment also includes generating cluster data representative of a cluster of anomalies from among the set of anomalies, where the cluster is formed using a lattice clustering algorithm that spatially distinguishes the cluster of anomalies from other anomalies in the set of anomalies. The embodiment also includes composing an explanation using log templates generated from log entries associated with the cluster of anomalies.