Hierarchical Log Entry Clustering for Analysis Efficiency
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Conventional solutions for processing log files or log entries are inefficient in analyzing the data, failing to extract valuable information and perform tasks like log data deduplication, workload prediction, failure cause analysis, and tracing on storage systems.
Innovation Solution
A method to obtain a hierarchical data structure by extracting characteristic information from log entries, clustering them based on patterns, and storing them in a hierarchical organization, enabling efficient storage and retrieval, as well as analytical processing.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Productivity
If conventional solutions are used to process log files, then the processing method is simple, but the analysis efficiency is low and valuable information cannot be extracted
Solution Approach 1:
The patent segments log entries into hierarchical groups based on extracted characteristics and patterns. Log entries are organized into parent-child relationships where parent nodes represent pattern groups and child nodes represent individual log entries or sub-groups, enabling efficient segmented analysis of large log datasets
Solution Approach 2:
The patent transforms flat log data into a multi-dimensional hierarchical structure by adding pattern-based classification dimensions. This creates multiple levels of organization (pattern level, group level, entry level) that enable analysis from different perspectives and improve processing efficiency
2Loss of time
If log entries are stored in a hierarchical structure with multiple nodes, then retrieval and analysis efficiency is improved, but the data structure becomes more complex
Solution Approach 1:
The patent performs preliminary actions by extracting characteristics and determining patterns before storing log entries. The hierarchical structure is pre-built with pattern-based grouping, so when analysis is needed, the work is already done and results can be retrieved quickly without complex real-time processing
Solution Approach 2:
The patent introduces pattern characteristics as intermediary elements between raw log entries and analysis queries. These pattern characteristics serve as mediators that enable efficient matching and retrieval by representing groups of similar log entries, reducing the complexity of direct entry-by-entry analysis
Data Source
AI summary
A method, an electronic device, and a computer program product for obtaining a hierarchical data structure and processing a log entry is disclosed. The method for obtaining the hierarchical data structure includes: obtaining corresponding characteristic information included in each log entry of a set of log entries and determining multiple log entry patterns based on the corresponding characteristic information. The pattern characteristic information of each log entry pattern corresponds to the characteristic information of a subset of log entries in the set of log entries. The method also includes storing the set of log entries according to the hierarchical data structure so that each log entry is associated with at least one of multiple nodes of the hierarchical data structure. The multiple nodes respectively correspond to the multiple log entry patterns, and are hierarchically organized based on respective corresponding log entry patterns.