Event-Based Search Support System for Flexible Data Extraction
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Modern data centers face challenges in analyzing massive volumes of heterogeneous performance data due to the unstructured nature of the data and the difficulty of applying semantic meaning, indexing, and querying, which leads to inefficiencies in data storage and retrieval.
Innovation Solution
The use of an event-based search support system like SPLUNK ENTERPRISE, which employs a late-binding schema and extraction rules to facilitate the identification of field values based on user-selected delimiters, allowing for flexible data processing and retrieval at search time.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Quantity of substance
If data is pre-processed by extracting and storing only some data items, then storage space is saved in the short term, but data availability is reduced in the longer term
Solution Approach 1:
The system performs preliminary indexing of all performance data at ingestion time without full processing, creating a searchable structure that enables later retrieval of any data item. This preliminary action preserves all data while enabling efficient access, resolving the contradiction between storage efficiency and data availability.
Solution Approach 2:
The system extracts only the essential indexing information from raw performance data at ingestion time, storing minimal processed data while preserving the ability to retrieve and analyze complete data items later. This extraction approach saves storage space while maintaining data availability through on-demand processing.
2Adaptability or versatility
If all performance data is stored and processed at search time, then data analysis flexibility is improved, but processing time and computational resources increase
Solution Approach 1:
The system performs preliminary indexing of all performance data at ingestion time, organizing data into a searchable structure that enables efficient later retrieval. This preliminary action reduces processing time at search time while maintaining the flexibility to analyze any data item, resolving the contradiction between adaptability and processing time.
3Productivity
If data is structured with pre-specified fields, then data retrieval efficiency is improved, but adaptability to investigate various aspects of performance data is reduced
Solution Approach 1:
The system uses a dynamic field extraction approach where field definitions are not fixed at ingestion time but are determined at search time based on user needs. This dynamic approach maintains efficient retrieval by using indexed data while providing flexibility to investigate any aspect of performance data, resolving the contradiction between productivity and adaptability.
Data Source
AI summary
In embodiments, method and systems are provided for facilitating identification of field values based on an extraction rule. In some implementations, a graphical user interface receives first input identifying an extraction mode that defines at least a portion of how to extract values from fields of events, and further receives second input configuring an extraction rule that defines at least a first field of the fields based on the extraction mode. The second input may include selecting a delimiter type for a delimiter mode, or specifying fields from a sample event for a regular expression mode. As such, an extraction rule may be configured, and a first set of the values from the events may be assigned to the first field based on the extraction rule.