Automated Log Analysis for Cloud Transaction Clustering

Resolve Bottlenecks,
Find Innovative SolutionsGenerate Solutions

Solution Overview

Problem

Cloud services face challenges in managing and analyzing vast volumes of log data from multiple sources without prior knowledge of transaction identifiers, making it difficult to detect meaningful trends and allocate resources effectively.

Innovation Solution

An automated log analysis method that processes log streams by pre-processing, filtering, indexing, and modeling transaction sequences, allowing for the detection of stateful event sequences without requiring a priori knowledge of transaction identifiers, and providing a transaction model for tracking and trending.

Engineering Contradictions & Design Principles

VSEngineering Contradiction Analysis

1Adaptability or versatility

If automated log analysis is implemented without prior knowledge of transaction identifiers, then the system can handle diverse transaction types and multiple log sources, but the complexity of detecting meaningful trends increases

Engineering Contradiction:

Improveability to handle diverse transaction types and multiple log sourcesVSAvoidcomplexity of detecting meaningful trends

Core Design Contradiction:

Adaptability or versatilityVSDevice complexity

Solution Approach 1:

The system performs self-service by automatically discovering transaction identifiers through clustering algorithms without requiring manual configuration or prior knowledge. The log analysis engine autonomously processes diverse log sources, identifies patterns, and creates transaction models independently, eliminating the need for manual intervention while handling varied transaction types effectively

Inventive Principle:

Principle #25Self-service

Solution Approach 2:

The system changes parameters by dynamically adjusting clustering thresholds and analysis criteria based on the characteristics of different log sources. It adapts its detection parameters automatically to accommodate diverse transaction types and multiple log formats, transforming the approach from static configuration to dynamic parameter adjustment

Inventive Principle:

Principle #35Parameter changes

2Measurement precision

If manual analysis of log data is performed, then detailed inspection is possible, but the time required to process large volumes of log data increases significantly

Engineering Contradiction:

Improvedetailed inspection capabilityVSAvoidtime required to process large volumes of log data

Core Design Contradiction:

Measurement precisionVSLoss of time

Solution Approach 1:

The system replaces manual mechanical analysis with automated computational processing. The log analysis engine uses algorithms to perform detailed inspection of log data automatically, substituting human analysts with machine-based processing that maintains precision while dramatically reducing the time required to analyze large volumes of log data

Inventive Principle:

Principle #28Mechanics substitution (Replace mechanical system)

Solution Approach 2:

The system creates copies of log data structures and transaction patterns to enable parallel processing and efficient analysis. By replicating data structures and using multiple analysis instances, it achieves detailed inspection of comprehensive log volumes without the time constraints of sequential manual review

Inventive Principle:

Principle #26Copying

3Loss of information

If comprehensive log data from multiple sources is collected, then more complete transaction tracking is achieved, but the volume of data to be processed increases

Engineering Contradiction:

Improvecompleteness of transaction trackingVSAvoidvolume of log data to be processed

Core Design Contradiction:

Loss of informationVSQuantity of substance

Solution Approach 1:

The system extracts only the essential and relevant information from comprehensive log data using clustering algorithms. It identifies and extracts key transaction identifiers and meaningful patterns from multiple log sources, separating useful information from unnecessary data volume while maintaining complete transaction tracking capability

Inventive Principle:

Principle #2Taking out (Extraction)

Solution Approach 2:

The system segments log data into distinct clusters representing different transaction types and sources. By dividing the comprehensive log volume into manageable segments that can be processed independently and then reassembled, it achieves complete transaction tracking while reducing the computational burden of processing raw data volume

Inventive Principle:

Principle #1Segmentation

Data Source

PatentUS10769152B2Automated log analysis

Publication Date: 2020.09.08 CISCO TECHNOLOGY INC

AI summary

There is disclosed in an example a computer-implemented method of providing automated log analysis, including: receiving a log stream comprising a plurality of transaction log entries, the log entries comprising a time stamp, a component identification (ID), and a name value pair identifying a transaction; creating an index comprising mapping a key ID to a name value pair of a log entry; and selecting from the index a key ID having a relatively large number of repetitions. There is also disclosed an apparatus and computer-readable medium for performing the method.