Universal Query Processing for Streaming and Batch Machine Data
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing data systems struggle to efficiently search and analyze large volumes of diverse, minimally processed machine data due to differences between data at rest and streaming data, leading to system complexity and inefficiencies in query processing.
Innovation Solution
A data intake and query system utilizing a flexible schema and late-binding schema to process machine data, enabling field-searchable events with extraction rules applied at search time, and supporting dynamic query processing based on data type and state.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Adaptability or versatility
If multiple query processors are used to handle different data types (data at rest and streaming data), then query processing capability is improved, but system complexity increases
Solution Approach 1:
The patent implements a universal query processor that can handle both data at rest and streaming data through a single system. The query processor dynamically adapts its behavior based on the input data type, using flexible schema validation and late-binding schema techniques to process diverse data formats without requiring separate specialized processors for each data type.
Solution Approach 2:
The query processor employs dynamic schema validation and late-binding schema approaches that allow the system to adapt its processing logic in real-time based on the characteristics of the incoming data. This dynamic adaptation enables the single processor to efficiently handle different data types (batch and streaming) without compromising performance or requiring static configuration for each data type.
2Adaptability or versatility
If raw machine data is stored for later analysis, then data flexibility and analysis capability are improved, but data storage requirements increase
Solution Approach 1:
The patent applies local quality by storing raw machine data with selective metadata and validation rules applied only where needed. Rather than processing or validating all data uniformly, the system applies schema validation and extraction rules selectively based on the specific analysis needs and data types, allowing efficient storage of diverse raw data while maintaining queryability only for relevant portions.
Solution Approach 2:
The system performs preliminary actions by implementing flexible schema validation and late-binding schema techniques during data ingestion, but defers detailed validation and processing until query time. This allows raw data to be stored with minimal processing overhead while ensuring data quality and consistency when actually queried, balancing storage efficiency with data integrity.
3Productivity
If field-searchable events with extraction rules are implemented, then data search efficiency is improved, but data processing complexity increases
Solution Approach 1:
The patent implements preliminary action by pre-defining extraction rules and flexible schema structures during data ingestion, but defers the actual application of these rules until query time through late-binding schema techniques. This allows the system to prepare the groundwork for efficient searching without immediately incurring the full processing complexity, enabling fast field-based searches when needed while keeping the ingestion process simpler.
Solution Approach 2:
The system uses an intermediary flexible schema layer that sits between the raw data and the search query processing. This intermediary layer handles the complexity of extraction rules and field validation, translating complex data structures into searchable formats without requiring complex processing logic in the search engine itself, thus improving search efficiency while isolating processing complexity in a manageable intermediate layer.
Data Source
AI summary
Operational machine components of an information technology (IT) or other microprocessor- or microcontroller-permeated environment generate disparate forms of machine data. Network connections are established between these components and processors of data intake and query system (DIQS). The DIQS conducts network transactions on a periodic and/or continuous basis with the machine components to receive disparate data and ingest certain of the data as entries of a data store that is searchable for DIQS query processing. The DIQS may receive queries to process against the received and ingested data via an exposed network interface. In one example embodiment, the DIQS receives a query identifying data to be processed, dynamically generates a query processing scheme based on the state of the data to be processed, such as streaming or at rest, and dynamically communicates the query processing scheme to a query executor based on the state of the data to be processed.