Indexer Bypass Search Head for Analytics Query Transmission

Resolve Bottlenecks,
Find Innovative SolutionsGenerate Solutions

Solution Overview

Problem

In data processing systems, bandwidth limitations often arise when data from multiple sources is processed through a single component, leading to bottlenecks that hinder efficient data processing and analysis, particularly in large-scale data centers and networks.

Innovation Solution

The implementation of a data intake and query system where indexers bypass the search head to transmit events directly to the analytics system, allowing for aggregation of responses and managing bandwidth by distributing computational operations across multiple indexers, thereby minimizing data transfers and optimizing query processing.

Engineering Contradictions & Design Principles

VSEngineering Contradiction Analysis

1Ease of operation

If data from multiple producers is processed through a single component (search head), then data processing can be centralized and coordinated, but bandwidth limitations and bottlenecks occur

Engineering Contradiction:

Improvecentralized coordinationVSAvoiddata processing throughput

Core Design Contradiction:

Ease of operationVSProductivity

Solution Approach 1:

The patent segments the data transmission path by allowing multiple indexers to transmit events directly to the analytics system in parallel, rather than funneling all data through a single search head. This segmentation creates multiple independent data flow channels, eliminating the bottleneck while maintaining coordinated processing through the analytics system's ability to handle multiple input streams simultaneously.

Inventive Principle:

Principle #1Segmentation

2Ease of operation

If indexers transmit events through the search head to the analytics system, then centralized query management is maintained, but bandwidth is limited and response time increases

Engineering Contradiction:

Improvequery managementVSAvoidquery response time

Core Design Contradiction:

Ease of operationVSLoss of time

Solution Approach 1:

The analytics system acts as an intermediary that can directly receive events from multiple indexers without requiring the search head as an intermediate relay. This eliminates the unnecessary routing step through the search head for analytics queries, reducing latency while the analytics system itself maintains the coordination function by processing and correlating events from multiple sources.

Inventive Principle:

Principle #24Intermediary (Mediator)

3Device complexity

If a single component processes all data, then system architecture is simplified, but bandwidth acts as a limit on processing ability

Engineering Contradiction:

Improvesystem architectureVSAvoiddata processing capacity

Core Design Contradiction:

Device complexityVSProductivity

Solution Approach 1:

The system architecture is segmented into multiple parallel processing paths where each indexer can independently transmit data to the analytics system. This segmentation increases processing capacity by utilizing aggregate bandwidth across multiple network paths while the analytics system's unified event processing logic maintains architectural simplicity in terms of coordination requirements.

Inventive Principle:

Principle #1Segmentation

Data Source

PatentUS11681707B1Analytics query response transmission

Publication Date: 2023.06.20 CISCO TECHNOLOGY INC

AI summary

Transmission handling of analytics query response includes a search head, in a data intake and query system, receiving a query from an analytics system. The search head distributes at least a portion of the query to at least one indexer for processing the query. The at least one indexer transmits, bypassing the search head, and to the analytics system, events matching the query. The search head receives from the at least one indexer, data regarding the events, and sends the data regarding the events to the analytics system.