Size-Based Event Groups for Message Bus Indexing
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Analyzing and searching massive quantities of machine data from diverse sources is time-consuming and inefficient due to varying data formats and the need for flexible schema application at search time.
Innovation Solution
A data intake and query system utilizing a late-binding schema and a message bus architecture with ingestors and indexers to dynamically scale event processing and indexing, enabling flexible schema application and improved throughput, resiliency, and load balancing.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Adaptability or versatility
If data is processed and indexed before storage using traditional schema approaches, then data can be efficiently queried, but the system cannot handle diverse data formats flexibly and requires extensive preprocessing
Solution Approach 1:
The patent applies preliminary action by storing data in a standardized format with embedded schema information before querying. The schema is prepared and attached to the data during ingestion, allowing the system to handle diverse formats flexibly without complex preprocessing at query time. This resolves the contradiction by preparing the schema application in advance rather than requiring complex runtime processing.
2Quantity of substance
If massive quantities of machine data are collected and stored, then comprehensive analysis is enabled, but analysis and searching become time-consuming
Solution Approach 1:
The patent segments the data processing system into multiple independent indexers that can process data in parallel. Each indexer handles a portion of the data stream, applying schema and indexing independently. This segmentation enables the system to handle massive quantities of machine data while maintaining high processing throughput, as the workload is distributed across multiple processing units rather than a single bottleneck.
Solution Approach 2:
The patent implements continuous data ingestion and processing through a stream processing architecture. Data is continuously collected, processed, and indexed without interruption, allowing the system to handle large volumes of machine data at high speed. The continuous processing pipeline ensures that data analysis productivity is maintained even as data volume increases.
3Productivity
If data is minimally processed during ingestion, then storage efficiency is improved, but data quality and consistency may be compromised
Solution Approach 1:
The patent applies local quality by processing and validating data locally at each indexer node during ingestion. Each indexer applies the appropriate schema to its assigned data stream, ensuring data quality and consistency at the source rather than requiring centralized validation. This local processing maintains high ingestion throughput while ensuring data reliability through distributed schema enforcement.
Data Source
AI summary
A data intake and query system can ingest and index large amounts of data using one or more ingestors and indexers. The ingestors can ingest incoming data, use it to generate events, and communicate the generated events to a message bus. Indexers can monitor their capacity to process additional groups of events. Based on a determination that an indexer has capacity to process one or more groups of events, the indexer can request one or more messages from the message bus, and process the events associated with the one or more messages.