Sankey Diagram Visualization for Machine Data Search Analysis
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Analyzing and searching massive quantities of machine-generated data poses challenges due to its vast volume and varied formats, with existing tools often discarding non-preprocessed data and limiting analysis flexibility.
Innovation Solution
The implementation of an event-based data intake and query system, such as the SPLUNK® ENTERPRISE system, which stores minimally processed machine data for later retrieval, using a late-binding schema to facilitate flexible data analysis and visualization, and a visualization framework that enables interactive visualizations based on static libraries, allowing users to interact with and modify visual representations of data.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Adaptability or versatility
If massive quantities of machine data are stored for later retrieval and analysis, then data analysis flexibility is improved, but data processing complexity increases
Solution Approach 1:
The system segments machine data into discrete events with structured fields, organizing vast quantities of raw data into manageable, queryable units. Each event contains specific data items that can be independently searched and analyzed, reducing the complexity of processing massive data volumes while maintaining analysis flexibility.
Solution Approach 2:
The patent introduces an intermediate representation layer between raw machine data and analysis tools. Events serve as this intermediary, converting unstructured machine data into a standardized format with defined fields and data types, enabling flexible querying without directly handling the complexity of raw data volumes.
2Speed
If pre-specified data items are extracted and stored during pre-processing, then data retrieval efficiency is improved, but data analysis flexibility deteriorates
Solution Approach 1:
The system performs preliminary organization of machine data into events with standardized fields during data ingestion, but maintains all original data items available for retrieval. This preliminary structuring enables efficient querying without pre-filtering data items, allowing analysts to retrieve any data item on-demand while benefiting from organized storage.
Solution Approach 2:
The event structure serves multiple functions simultaneously: it organizes data for efficient storage, enables rapid retrieval through field-based indexing, and supports flexible analysis by preserving all original data items. This universal structure replaces the need for separate pre-processing pipelines for different analysis needs.
3Loss of information
If minimally processed machine data is stored instead of pre-processed data, then data completeness is improved, but search and analysis performance deteriorates
Solution Approach 1:
The system segments minimally processed machine data into structured events with defined fields, maintaining data completeness while enabling efficient search. Each event contains all original data items organized by field, allowing the system to preserve complete information while providing structured access paths for high-performance retrieval and analysis.
Data Source
AI summary
Disclosed are a system and a method for providing user-interactive flow diagrams such as Sankey diagrams. In an embodiment, a machine data search and analysis system retrieves search results including a plurality of events in response to a search query, each of the events containing time-stamped data. The system identifies a plurality of nodes based on the time-stamped data of the events. Each of the events is indicative of a relationship between a starting node and an ending node. The system further causes display of an interactive flow diagram that includes the nodes and a plurality of flows. Each of the flows represents events that share a common relationship between a starting node and an ending node and visually connecting the starting node and the ending node.