Event-Based Data Intake System for Flexible Machine Data Querying
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Analyzing and searching massive quantities of machine data presents challenges due to its vastness and diversity, requiring efficient methods to manage, understand, and utilize this data effectively.
Innovation Solution
An event-based data intake and query system processes, indexes, and stores machine data as events, allowing for flexible schema application at search time, enabling field-searchable capabilities and efficient querying across diverse data sources.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Adaptability or versatility
If massive quantities of raw machine data are stored for later retrieval and analysis, then data analysis flexibility and completeness are improved, but data storage requirements and retrieval complexity increase
Solution Approach 1:
The patent segments machine data into discrete events with structured fields, organizing massive data volumes into manageable, queryable units. Each event contains specific fields (timestamp, host, source, sourcetype, etc.) that can be independently indexed and searched, transforming unwieldy raw data into organized, analysis-ready information while maintaining storage efficiency through selective field extraction and indexing.
2Adaptability or versatility
If diverse data from numerous devices is collected and stored, then data diversity and insight potential are improved, but data management and search complexity increase
Solution Approach 1:
The patent implements a universal event schema that can accommodate diverse machine data from numerous devices through a common structure. The event format with standardized fields (timestamp, host, source, sourcetype, etc.) provides a multi-functional framework that handles various data types uniformly, enabling consistent management and search across heterogeneous data sources while preserving data diversity.
Solution Approach 2:
The patent applies local quality by allowing specific field structures and data types to vary according to the data source and context while maintaining an overall uniform event framework. Different devices can contribute events with source-specific field variations, enabling tailored data capture for each device type while maintaining global consistency through the common event schema and standardized indexing approach.
3Productivity
If pre-processing is applied to reduce data volume, then data retrieval efficiency is improved, but data analysis flexibility and completeness deteriorate
Solution Approach 1:
The patent performs preliminary action by structuring and indexing key fields during data ingestion without discarding raw data. The system pre-processes data to extract and index essential fields (timestamp, host, source, sourcetype) for efficient retrieval, while maintaining the complete event structure for later comprehensive analysis. This preliminary organization enables fast search while preserving all original data for complete analysis when needed.
Data Source
AI summary
Systems and methods are described for generation and execution of modified queries. An input can be received via a visualization of a user interface. The input may identify a first field value and a first field for execution of a query. A set of data for execution of the query can be identified based on the input. Alias data may identify a second field that is associated with the first field. Using the alias data, a modified query can be generated based on the query and the second field. The modified query can be executed to generate query results. The query results can be displayed via a visualization of the user interface based on the first field.