Event Stream GUI Configuration for Network Performance Analysis
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Analyzing massive quantities of heterogeneous performance data at search time is challenging due to the unstructured nature of the data and the difficulty of indexing and querying it using traditional database systems, especially in data centers that generate significant volumes of performance data from thousands of different components.
Innovation Solution
Utilizing an event-based system like the SPLUNK® ENTERPRISE system to store and process performance data, employing a late-binding schema that allows flexible schema development at search time, enabling extraction rules to be applied on the fly, and using parallel processing techniques to manage and analyze the data efficiently.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Productivity
If traditional database systems are used to store and query performance data, then data storage and retrieval infrastructure is simple, but analysis of massive heterogeneous performance data becomes challenging due to unstructured nature and difficulty of indexing
Solution Approach 1:
The system segments performance data into discrete events with structured fields, transforming unstructured data into indexable units. Each event is divided into key-value pairs that can be independently indexed and queried, enabling efficient analysis of massive heterogeneous data volumes.
Solution Approach 2:
The system changes the data representation parameters from traditional relational tables to event-driven key-value pairs with flexible schemas. This allows the same infrastructure to handle both structured and unstructured data uniformly, improving analysis capability without proportionally increasing complexity.
2Adaptability or versatility
If pre-specified schema is used for data storage, then data indexing and querying is efficient, but flexibility to investigate different aspects of data at search time is reduced
Solution Approach 1:
The system performs preliminary action by collecting and normalizing data into standardized event formats at the time of data generation. This preparation enables flexible queries at search time without requiring complex real-time processing, as the data structure is already optimized for indexing while maintaining schema flexibility.
Solution Approach 2:
The system implements dynamic schema handling where the schema is not fixed beforehand but can adapt to different query requirements. The same event structure can be queried in multiple ways, and new fields can be added without affecting existing data, providing both flexibility and efficiency.
3Loss of information
If massive performance data is stored for later analysis, then data availability for investigation is improved, but storage requirements and data management complexity increase significantly
Solution Approach 1:
The system extracts only the essential information from performance data into standardized event fields, separating meaningful data from redundant information. This extraction reduces the effective data volume that needs to be stored while maintaining all necessary information for analysis, as only relevant fields are captured and indexed.
Solution Approach 2:
The system creates a universal event format that can store multiple types of performance data from different sources in a unified structure. This multi-functionality allows diverse data to be stored using the same infrastructure and indexing mechanisms, reducing overall storage requirements compared to separate storage systems for different data types.
4Adaptability or versatility
If remote capture agents are deployed across the network to collect data, then data collection coverage is improved, but network infrastructure complexity and data transmission overhead increase
Solution Approach 1:
The remote capture agents perform self-service by automatically collecting, parsing, and normalizing data locally before transmission. Each agent independently processes data according to standardized protocols without requiring complex centralized coordination, reducing network infrastructure complexity while maintaining broad data collection capability across the network.
Data Source
AI summary
Disclosed embodiments receive an event stream from a remote capture agent. The event stream includes timestamped event data generated by the remote capture agent based on network traffic monitored by the remote capture agent. A graphical user interface (GUI) is caused to be displayed for obtaining configuration information for configuring the generation of time-series event data from network packets captured by the remote capture agent. A set of statistics is generated from the time-series event data, and the configuration information is updated to trigger subsequent storage and processing of at least a portion of the event stream by one or more components on a network based on one or more of the statistics, a storage limit associated with the time-series event data, an index volume of the event stream, a historical trend associated with the statistics, or input through the GUI.