Containerized Search Nodes for Scalable Data Querying
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Current tools lack the ability to efficiently search and analyze large sets of raw machine data from diverse sources, as they often discard unprocessed data during pre-processing, limiting the flexibility and insight that can be gained from analyzing all generated data.
Innovation Solution
A data intake and query system that utilizes a late-binding schema and containerized indexing and search nodes to process, index, and store data in a common storage system, allowing for flexible querying and analysis of minimally processed machine data.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Speed
If data is pre-processed and filtered during data ingestion, then search and analysis speed is improved, but data flexibility and completeness are lost
Solution Approach 1:
The system segments the data processing workflow into distinct stages: data ingestion, indexing, and querying. Raw data is ingested and minimally processed into events with extracted fields, then indexed in a way that preserves the original data while enabling efficient search. This segmentation allows the system to maintain raw data for flexibility while creating structured indexes for fast retrieval.
Solution Approach 2:
The system performs preliminary actions during data ingestion by extracting key fields from raw data and creating indexes, but does not discard or heavily process the original data. Events are created with extracted fields that enable efficient querying, while the raw data remains intact in the indexed format, allowing both fast search and data flexibility.
2Quantity of substance
If all raw machine data is stored for later retrieval, then data completeness and analysis flexibility are improved, but storage requirements and data retrieval complexity increase
Solution Approach 1:
The system segments data into events with extracted fields during ingestion, creating a structured representation that is easier to retrieve and analyze. Each event contains relevant fields extracted from the raw data, organizing the information in a way that reduces retrieval complexity while preserving data completeness.
Solution Approach 2:
The system introduces events as an intermediary representation between raw machine data and query operations. Events contain extracted fields that serve as a structured interface for querying, simplifying data retrieval while maintaining access to all original data through the event structure.
3Adaptability or versatility
If data is minimally processed during ingestion, then data flexibility and completeness are improved, but search and analysis efficiency decrease
Solution Approach 1:
The system performs preliminary field extraction and event creation during data ingestion, organizing data into a structured format with extracted fields. This preliminary action enables efficient search and analysis operations without requiring heavy processing of the original data, maintaining flexibility while improving productivity.
Solution Approach 2:
The system segments raw data into structured events with extracted fields during ingestion. This segmentation creates an organized representation that enables efficient querying and analysis operations while preserving the flexibility to work with all original data, resolving the contradiction between data flexibility and search efficiency.
Data Source
AI summary
Systems and methods are disclosed for processing and executing queries in a data intake and query system. The query identifies a set of data to be processed and a manner of processing the set of data. The data intake and query system dynamically identifies a plurality of containerized search nodes instantiated on one or more computing devices in a containerized environment to execute the query. The data intake and query system executes the query using the containerized search nodes.