Dynamic Field Extraction from Unstructured Log Data
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
In large-scale virtualized computing infrastructures, identifying and debugging failures and performance issues is challenging due to the overwhelming volume and unstructured nature of log data generated by numerous software and hardware components.
Innovation Solution
A graphical user interface is provided for analyzing log messages, allowing users to dynamically extract and define fields from log text, which are then used for searches, filters, charts, and statistical analysis, with live highlighting to assist in modifying definitions and visual feedback.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Loss of information
If log data is collected from all components of virtualized computing infrastructure, then complete troubleshooting information is obtained, but the volume and complexity of log data becomes overwhelming
Solution Approach 1:
The patent extracts only the relevant log entries related to failures and performance issues from the massive volume of log data generated by virtualized computing infrastructure. The system identifies and isolates specific log messages that contain diagnostic information, separating them from the overwhelming majority of unrelated log data.
Solution Approach 2:
The patent introduces an intermediary processing layer that automatically analyzes, filters, and prioritizes log data before presentation to users. This intermediary system uses pattern matching and contextual analysis to bridge the gap between raw log data and actionable troubleshooting information.
2Measurement precision
If log data is analyzed manually to identify problems, then detailed inspection is possible, but the time and effort required becomes prohibitive
Solution Approach 1:
The patent performs preliminary automated analysis of log data, pre-identifying potential issues and organizing relevant information before the user begins troubleshooting. Log entries are pre-filtered, pre-categorized, and pre-prioritized based on their diagnostic value and relationship to known failure patterns.
Solution Approach 2:
The patent implements feedback mechanisms where the system learns from user interactions with log data, automatically improving its filtering and prioritization algorithms. The system provides feedback about identified patterns and allows users to refine the analysis, creating a continuous improvement loop that reduces analysis time while maintaining accuracy.
3Measurement precision
If complex programming is required to extract fields from log text, then precise data extraction is achieved, but the ease of operation deteriorates
Solution Approach 1:
The patent enables self-service field extraction where the system automatically identifies and extracts relevant fields from log text without requiring users to write complex programming code. The system uses natural language processing and pattern recognition to automatically parse log entries and extract meaningful data fields.
Solution Approach 2:
The patent creates a universal field extraction mechanism that handles multiple log formats and structures through a single interface. The system is designed to work with various types of log data from different components of the virtualized infrastructure without requiring format-specific programming, making the extraction process both precise and accessible to non-programmers.
Data Source
AI summary
A log analytics graphical user interface enables a user to dynamically extract and define a field from unstructured log data. The log analytics module automatically determines a definition for a field based on log text selected by the user. A portion of each log message is highlighted to reflect what the extracted field may be to assist users with understanding if input parameters are selected the intended log data. Changes to the definition of the field, by the user, may cause further highlighting that to indicate an incomplete or erroneous field definition.