Dual Textual Graphical Interfaces for Streaming Data Pipelines
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Current tools lack the ability to efficiently and flexibly analyze and search large quantities of raw machine data from diverse sources, as they typically pre-process data based on anticipated needs, discarding the rest and not providing straightforward, user-friendly interfaces for visual identification of data subsets.
Innovation Solution
A data intake and query system that uses a flexible schema to process and store raw machine data, allowing for late-binding schema application during search time, enabling field-searchable events and facilitating the use of a common information model across disparate data sources through indexing and querying mechanisms.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Productivity
If data systems pre-process data based on anticipated analysis needs, then data retrieval efficiency is improved, but data flexibility and completeness are worsened because most generated data is discarded
Solution Approach 1:
The system performs preliminary indexing of all raw machine data without pre-processing or filtering, creating searchable event structures in advance. This allows the data to be ready for retrieval while maintaining complete flexibility for any future analysis needs, resolving the contradiction between preparation efficiency and adaptability.
Solution Approach 2:
The system changes the parameter of data processing from pre-processing specific data items to indexing all data items with event structures. This parameter change enables the system to maintain all raw data while providing efficient retrieval capabilities for any analysis scenario, eliminating the need to discard data.
2Adaptability or versatility
If all raw machine data is stored for later analysis, then data analysis flexibility is improved, but data search and analysis complexity increases
Solution Approach 1:
The system segments all raw machine data into discrete events with standardized structures, where each event contains specific fields and properties. This segmentation transforms the complexity of searching unstructured raw data into efficient querying of structured events, maintaining flexibility while reducing search complexity.
Solution Approach 2:
The system introduces an event structure as an intermediary between raw machine data and analysis queries. This event structure serves as a standardized format that simplifies searching and analysis while preserving all original data, acting as a mediator that reduces complexity without sacrificing flexibility.
3Loss of information
If tools provide comprehensive data search capabilities, then data analysis completeness is improved, but user interface complexity and ease of use deteriorates
Solution Approach 1:
The system creates simplified copies or representations of the comprehensive data through event summaries, thumbnails, and aggregated views that can be displayed in user-friendly interfaces. These copies provide complete data access capabilities while presenting information in easily consumable formats, resolving the contradiction between completeness and usability.
Data Source
AI summary
Systems and methods are disclosed for implementing dual textual/graphical programming interfaces for programming streaming data processing pipelines. A user interface is provided that enables a user to author a processing pipeline as a query in a query language, and to request conversion of that query language into a graph data structure representation of the pipeline, which can be visualized in the interface. The interface further enables modification of the graph via interaction with the visualization. On request, the modified graph can be converted back into the query language, with the querying being modified to reflect the modifications to the graph.