Wizard-Guided Field Extraction for Evolving Machine Data
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
The challenge lies in formulating and refining field extraction rules for machine-generated data, which often lacks a defined format, making it difficult to develop schemas and extraction rules, especially since the data format is not determined at the time of collection and continues to evolve.
Innovation Solution
A wizard-guided process that combines analytical interfaces to help users select source types, example events, fields to extract, validate extraction results, and save extraction rules, allowing for the formulation of rules that can handle multiple data formats through tools that support positive and negative examples, sampling, and validation.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Adaptability or versatility
If traditional schema development methods are used for machine-generated data, then the process becomes complex and time-consuming, but the data format is not determined at collection and continues to evolve
Solution Approach 1:
The system performs preliminary analysis of data samples to automatically generate extraction rules before full-scale data processing begins. This preliminary action establishes the schema framework in advance, reducing complexity during ongoing operations while maintaining adaptability to format changes through iterative rule refinement.
Solution Approach 2:
The schema development system performs self-service by automatically analyzing data samples, identifying patterns, and generating extraction rules without requiring extensive manual configuration. The system serves itself by continuously refining rules based on new data formats, reducing the need for human intervention while adapting to evolving data structures.
2Measurement precision
If manual extraction rule formulation is used, then precision can be achieved, but the process requires extensive programming and time
Solution Approach 1:
The system creates extraction rules by copying and adapting patterns from analyzed data samples rather than requiring manual programming from scratch. This copying approach maintains precision by learning actual data patterns while dramatically reducing development time through automated pattern recognition and rule generation.
Solution Approach 2:
The system implements feedback loops where extraction rules are continuously tested against new data samples, with performance metrics fed back into the rule refinement process. This feedback mechanism maintains high precision by automatically adjusting rules based on actual performance while reducing manual intervention time through iterative self-optimization.
3Reliability
If comprehensive validation is performed on extraction rules, then reliability improves, but the processing time increases
Solution Approach 1:
The system applies partial validation by focusing comprehensive checking on critical extraction rules while using lighter validation for less critical ones. This selective approach maintains reliability for important data fields while preserving productivity by avoiding excessive validation overhead on all rules uniformly.
Solution Approach 2:
Comprehensive validation is performed as a preliminary action during the rule generation phase rather than during ongoing operations. This upfront validation ensures reliability is established before deployment while maintaining productivity by avoiding repeated validation cycles during normal data processing.
Data Source
AI summary
The technology disclosed relates to formulating and refining field extraction rules that are used at query time on raw data with a late-binding schema. The field extraction rules identify portions of the raw data, as well as their data types and hierarchical relationships. These extraction rules are executed against very large data sets not organized into relational structures that have not been processed by standard extraction or transformation methods. By using sample events, a focus on primary and secondary example events help formulate either a single extraction rule spanning multiple data formats, or multiple rules directed to distinct formats. Selection tools mark up the example events to indicate positive examples for the extraction rules, and to identify negative examples to avoid mistaken value selection. The extraction rules can be saved for query-time use, and can be incorporated into a data model for sets and subsets of event data.