Extraction Rule Validation via Match Rate Analysis
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Analyzing and searching massive quantities of machine-generated data from diverse sources is challenging due to the time-consuming nature of processing various types and formats of data from thousands of components, such as system logs, network packets, and sensor data, which existing systems struggle to handle efficiently.
Innovation Solution
The implementation of an event-based data intake and query system, like the SPLUNK® ENTERPRISE system, that uses a late-binding schema to extract values for specific fields during search time through extraction rules, allowing for flexible data analysis and storage of minimally processed data for later retrieval and analysis.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Adaptability or versatility
If extraction rules are used to extract data from machine-generated data, then data analysis capability is improved, but processing time increases
Solution Approach 1:
The system performs preliminary actions by creating keyword indexes from extracted fields during data ingestion. This allows the index to be pre-built with structured data, so that during query operations, the system can quickly retrieve results without performing full extraction rules processing at query time, thus reducing processing time while maintaining analysis capability
Solution Approach 2:
The system segments data processing into two phases: ingestion time extraction (building keyword indexes with selected fields) and query time retrieval (using pre-built indexes). This segmentation allows comprehensive data analysis capability to be maintained through thorough extraction during ingestion, while query processing speed is improved by using pre-processed keyword indexes rather than re-applying extraction rules
2Loss of information
If all machine data is stored for later retrieval, then data completeness is improved, but storage requirements increase
Solution Approach 1:
The system extracts only the most relevant fields from machine data during ingestion and creates keyword indexes based on these extracted fields. This selective extraction maintains data completeness for analysis purposes while reducing storage requirements by not storing all raw data in processed form, only storing the extracted keyword indexes that are essential for querying
3Measurement precision
If complex extraction rules are applied to all data, then data accuracy is improved, but processing speed decreases
Solution Approach 1:
Complex extraction rules are applied in advance during data ingestion to build accurate keyword indexes. This preliminary application of extraction rules ensures data accuracy is achieved during the ingestion phase, while query operations benefit from pre-processed indexes that do not require re-applying complex rules, thus maintaining high processing speed during retrieval
Solution Approach 2:
The system applies extraction rules selectively to create keyword indexes for the most frequently queried fields during ingestion, rather than processing all possible data transformations for all data. This partial application of extraction rules maintains sufficient data accuracy for common queries while significantly improving processing speed by avoiding excessive processing of less important data aspects
Data Source
AI summary
Embodiments of the present invention are directed to validating extraction rules. In embodiments, a set of events for which field extraction is desired is obtained. Thereafter, an extraction rule is applied to the set of events to extract fields of the events. The application of the extraction rule can be monitored to determine that the applied extraction rule is invalid. Based on the applied extraction rule being invalid, a new extraction rule can be generated to apply to the set of events.