Containerized State-Free Indexing Nodes for Scalable Data Querying
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Current tools lack the ability to efficiently and flexibly search and analyze large sets of raw machine data from diverse sources, as they often discard unprocessed data and require pre-defined schemas, limiting the ability to derive insights from all generated data.
Innovation Solution
A data intake and query system that utilizes a late-binding schema, allowing for flexible data processing and storage of raw machine data, enabling efficient retrieval and analysis by indexing and querying machine data as events with timestamped field-searchability, and utilizing containerized indexing and search nodes for scalable query execution.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Speed
If pre-processing data to extract specified items is performed, then retrieval efficiency is improved, but data loss increases as remainder data is discarded
Solution Approach 1:
The system performs preliminary indexing on specified data items during data ingestion, creating an inverted index that enables efficient retrieval without requiring pre-processing of all data. This allows the system to prepare search structures in advance while maintaining the ability to retrieve both indexed and unindexed data as needed.
Solution Approach 2:
The system discards the traditional approach of discarding unprocessed data during pre-processing. Instead, it recovers and preserves all raw data in its original form while creating selective indexes on specific fields. This allows the system to retrieve only indexed data when needed for efficiency, or access the complete raw data set when comprehensive analysis is required, thus eliminating permanent data loss.
2Device complexity
If pre-defined schemas are used for data processing, then data structure management is simplified, but adaptability decreases limiting insight derivation
Solution Approach 1:
The system implements dynamic schema evolution by allowing the data model to adapt automatically as new data types and structures are ingested. The schema is not fixed in advance but evolves to accommodate diverse data sources including logs, metrics, traces, and unstructured data, enabling the system to maintain simplicity while increasing adaptability over time.
Solution Approach 2:
The system creates a universal data model that can handle multiple data types (logs, metrics, traces, unstructured data) through a common event-based structure. This universal approach simplifies management by providing consistent handling across diverse data sources while simultaneously increasing adaptability to accommodate any new data type that fits the event model.
3Adaptability or versatility
If all raw machine data is stored for later retrieval, then data analysis flexibility is improved, but storage requirements increase
Solution Approach 1:
The system segments data storage into two distinct layers: an indexed layer containing processed and indexed data for efficient retrieval of specific fields, and a raw data layer preserving complete unprocessed data for comprehensive analysis. This segmentation allows the system to optimize storage by storing only essential indexed information in the first layer while maintaining full raw data capability in the second layer, thus reducing overall storage requirements while preserving analysis flexibility.
4Productivity
If containerized state-free indexing nodes are used, then system scalability is improved, but system complexity increases
Solution Approach 1:
The containerized indexing nodes are designed to be state-free and self-sufficient, automatically managing their own operation without requiring complex orchestration or state management infrastructure. Each node independently processes data, creates indexes, and manages its own lifecycle, eliminating the need for complex distributed state management systems and reducing overall system complexity while maintaining high scalability.
Data Source
AI summary
Systems and methods are disclosed for processing and executing queries in a data intake and query system. The data intake and query system receives raw machine data at an indexing system, and stores at least a portion of the raw machine data in buckets using containerized indexing nodes instantiated in a containerized environment. The data intake and query system stores the buckets in a shared storage system.