Log-to-Metrics Application for Automated Data Extraction
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Manual analysis of raw machine data for trend identification in IT environments is time-consuming, prone to errors, and requires specialized skills, leading to decreased productivity and imprecision in determining server performance metrics.
Innovation Solution
A log-to-metrics application that automates the extraction of measurement values from raw machine data by generating mappings between metric identifiers and field values, storing these mappings in a configuration file, and using them to extract and aggregate field values for further analysis.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Productivity
If manual analysis of raw machine data is performed, then flexibility in data exploration is maintained, but time consumption increases and productivity decreases
Solution Approach 1:
The system enables self-service by allowing users to define custom metrics through a graphical interface without requiring programming skills. The automated metric generation system performs the complex data extraction and aggregation tasks automatically, making the system serve itself rather than requiring manual intervention for routine analysis tasks.
Solution Approach 2:
The patent replaces the mechanical manual process of analyzing raw machine data with an automated computational system. Instead of manually parsing logs and calculating metrics, the system uses programmed algorithms to automatically extract, transform, and aggregate data, substituting human labor with automated mechanical processes.
2Measurement precision
If specialized programs or scripts are written to analyze data, then measurement precision can be improved, but device complexity and skill requirements increase
Solution Approach 1:
The system introduces an intermediary layer between the user and the complex data processing logic. This intermediary is the graphical user interface and configuration system that translates simple user definitions into complex automated processing tasks, hiding the underlying complexity while maintaining precision.
Solution Approach 2:
The system allows users to define metrics by changing parameters such as time ranges, aggregation functions, and field selections through a graphical interface. These parameter changes are automatically translated into executable queries, enabling precise metric extraction without requiring users to understand or write complex programming code.
3Reliability
If manual trend identification is performed, then adaptability to different data formats is maintained, but measurement precision and reliability decrease
Solution Approach 1:
The system performs preliminary action by pre-defining standard metrics and templates that can be automatically applied to various data formats. Users can select from pre-configured metric templates or define custom ones, and the system automatically generates the necessary extraction and aggregation logic before actual analysis begins, ensuring consistent and reliable results.
Data Source
AI summary
A log-to-metrics transformation system includes a log-to-metrics application executing on a processor. The log-to-metrics transformation system receives a format associated with machine data, and further receives, via a first graphical control, a first set of metric identifiers corresponding to a first set of metrics associated with the machine data. The log-to-metrics transformation system generates a first set of mappings between the first set of metric identifiers and a first set of field values included in the machine data. The log-to-metrics transformation system stores the first set of mappings and an association with the format of the machine data. The log-to-metrics transformation system, based on the first set of mappings, causes the first set of field values to be extracted from the machine data. Further, a first metric included in the first set of metrics is determined based on at least a portion of the first set of field values.