Clustering-Based Data Event Indexing for Unstructured Machine Logs

Resolve Bottlenecks,
Find Innovative SolutionsGenerate Solutions

Solution Overview

Problem

Modern data centers face challenges in efficiently processing and analyzing large volumes of heterogeneous machine-generated data due to the unstructured nature of the data, making it difficult to apply semantic meaning and perform effective indexing and searching operations.

Innovation Solution

The system employs a data aggregation and analysis system that performs real-time indexing and executes searches by generating search terms through preliminary grouping (clustering) of events, identifying relevant tokens, and determining search terms that can reproduce the grouping, allowing for efficient retrieval of meaningful data groups.

Engineering Contradictions & Design Principles

VSEngineering Contradiction Analysis

1Loss of information

If traditional indexing and searching operations are applied to unstructured machine-generated data, then semantic meaning can be extracted, but the processing complexity and resource requirements increase significantly

Engineering Contradiction:

Improvesemantic meaning extractionVSAvoidprocessing complexity

Core Design Contradiction:

Loss of informationVSDevice complexity

Solution Approach 1:

The system performs preliminary clustering of events into groups based on similarity metrics before indexing and searching. This pre-grouping action reduces the complexity of subsequent operations by organizing unstructured data into meaningful clusters, allowing faster retrieval without losing semantic information.

Inventive Principle:

Principle #10Preliminary action

Solution Approach 2:

The patent introduces cluster representatives as intermediary elements that summarize groups of similar events. These representatives serve as mediators between the raw unstructured data and the search/indexing operations, reducing processing complexity while preserving semantic meaning through condensed representations.

Inventive Principle:

Principle #24Intermediary (Mediator)

2Speed

If large volumes of machine-generated data are processed in real-time, then data retrieval speed improves, but resource consumption increases

Engineering Contradiction:

Improvedata retrieval speedVSAvoidresource consumption

Core Design Contradiction:

SpeedVSUse of energy by moving object

Solution Approach 1:

The system extracts only the essential characteristics of event groups through clustering, creating condensed representations that capture the semantic meaning. This extraction reduces the volume of data that needs to be processed and stored in detail, lowering resource consumption while maintaining fast retrieval speeds through efficient search on cluster representatives.

Inventive Principle:

Principle #2Taking out (Extraction)

3Loss of information

If unstructured data is indexed without preliminary grouping, then complete data is available for searching, but the searching operation becomes less efficient

Engineering Contradiction:

Improvedata completenessVSAvoidsearching efficiency

Core Design Contradiction:

Loss of informationVSProductivity

Solution Approach 1:

The system performs preliminary clustering to organize unstructured data into groups of similar events before indexing. This pre-processing maintains data completeness by preserving all original events while organizing them into meaningful clusters, enabling efficient searching through cluster representatives without losing access to complete data.

Inventive Principle:

Principle #10Preliminary action

Data Source

PatentUS11314733B2Identification of relevant data events by use of clustering

Publication Date: 2022.04.26 CISCO TECHNOLOGY INC

AI summary

A processing device performs a preliminary grouping of data items in a dataset to define one or more clusters and for each cluster, identifies a set of search terms for a search query that would retrieve data items in the cluster upon execution of the search query against the dataset.