Unified Query-Pipeline Conversion Across Streaming and Indexed Data
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Current tools lack the ability to efficiently search and analyze large sets of raw machine data from diverse sources, as they do not provide straightforward and easy-to-understand visual interfaces for identifying data subsets of interest.
Innovation Solution
A data intake and query system with a graphical programming system that allows for designing data processing pipelines visually, enabling unified processing of indexed and streaming data, and providing a flexible schema for extracting information from events, which facilitates searching and analyzing machine data.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Ease of operation
If traditional separate data search tools are used for different data systems, then data can be searched and collected, but the analysis must be done in a piecemeal manner without unified visual interface
Solution Approach 1:
The patent combines multiple separate data search and analysis tools into a single unified platform that can simultaneously search across diverse data systems (relational databases, cloud services, machine data sources) and present results in a unified visual interface, eliminating the need for piecemeal analysis across separate tools
Solution Approach 2:
The unified data processing platform performs multiple functions including searching diverse data formats, visualizing data relationships, analyzing patterns, and exporting results within a single system, replacing the need for multiple specialized tools while maintaining comprehensive data analysis capability
2Adaptability or versatility
If massive quantities of raw data are stored for later retrieval, then greater flexibility and completeness of analysis is enabled, but the complexity of searching and analyzing the data increases
Solution Approach 1:
The patent introduces an intermediary processing layer that sits between the stored raw data and the user interface, automatically parsing, normalizing, and indexing data from diverse sources into a unified structure that simplifies subsequent search and analysis operations without reducing analysis flexibility
Solution Approach 2:
The system performs preliminary data processing, validation, and indexing when data is first ingested and stored, preparing it in advance for efficient retrieval and analysis. This upfront processing reduces the complexity of subsequent search operations while maintaining the ability to analyze all raw data
Data Source
AI summary
Systems and methods are described for unified processing of indexed and streaming data. A system enables users to query indexed data or specify processing pipelines to be applied to streaming data. In some instances, a user may specify a query intended to be run against indexed data, but may specify criteria that includes not-yet-indexed data (e.g., a future time frame). The system may convert the query into a data processing pipeline applied to not-yet-indexed data, thus increasing the efficiency of the system. Similarly, in some instances, a user may specify a data processing pipeline to be applied to a data stream, but specify criteria including data items outside the data stream. For example, a user may wish to apply the pipeline retroactively, to data items that have already exited the data stream. The system can convert the pipeline into a query against indexed data to satisfy the users processing requirements.