Dynamic Query Processing for Streaming and Batch Machine Data
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing data systems face challenges in efficiently searching and analyzing large volumes of diverse machine data due to differences between data at rest and streaming data, leading to system complexity and inefficiencies in query processing.
Innovation Solution
A dynamic query processor that adapts its processing scheme based on the type and state of the data, using a late-binding schema to facilitate flexible and efficient querying of minimally processed machine data, enabling insights from all generated data.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Adaptability or versatility
If multiple query processors are used to handle different data types (data at rest and streaming data), then query processing capability is improved, but system complexity increases
Solution Approach 1:
The patent implements a universal query processor that can handle both data at rest and streaming data through a unified architecture. The system uses a single query processor that dynamically adapts its processing mode based on the input data type, eliminating the need for separate specialized processors and reducing system complexity while maintaining versatility
Solution Approach 2:
The query processor employs dynamic behavior by automatically adjusting its processing scheme based on the state of the underlying data. The system transitions between different processing modes (batch processing for data at rest, streaming processing for streaming data) without requiring multiple static processors, achieving adaptability through dynamic configuration rather than structural complexity
2Adaptability or versatility
If raw machine data is stored for later analysis, then data analysis flexibility is improved, but data retrieval and analysis efficiency decreases
Solution Approach 1:
The system performs preliminary actions by maintaining raw machine data in a readily accessible format with associated metadata and indexing structures. This allows the data to be retrieved and analyzed efficiently later without requiring extensive pre-processing or transformation, as the data is preserved in its original state with supporting infrastructure in place for quick access
Solution Approach 2:
The patent introduces an intermediary layer between raw data storage and analysis operations. This layer includes indexing mechanisms, metadata structures, and query optimization components that enable efficient retrieval and analysis of raw machine data without compromising the flexibility to analyze all generated data. The intermediary structures facilitate fast access while preserving data integrity and completeness
Data Source
AI summary
Operational machine components of an information technology (IT) or other microprocessor- or microcontroller-permeated environment generate disparate forms of machine data. Network connections are established between these components and processors of data intake and query system (DIQS). The DIQS conducts network transactions on a periodic and/or continuous basis with the machine components to receive disparate data and ingest certain of the data as entries of a data store that is searchable for DIQS query processing. The DIQS may receive queries to process against the received and ingested data via an exposed network interface. In one example embodiment, the DIQS receives a query identifying data to be processed, dynamically generates a query processing scheme based on the state of the data to be processed, such as streaming or at rest, and dynamically communicates the query processing scheme to a query executor based on the state of the data to be processed.