Unified Query and Pipeline Conversion Across Streaming Data
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Current tools lack efficient methods for quickly searching and analyzing large sets of raw machine data from diverse sources, such as system logs and network data, to identify relevant subsets, due to the complexity and volume of the data generated in IT environments.
Innovation Solution
A data intake and query system that utilizes a flexible schema to process and store raw machine data as events, allowing for field-searchable queries and late-binding schema application during search time, enabling the use of graphical programming for designing data processing pipelines and providing unified processing of indexed and streaming data.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Adaptability or versatility
If data systems store and pre-process large amounts of raw machine data from diverse sources, then data availability and analysis flexibility improve, but data complexity and search difficulty increase
Solution Approach 1:
The system segments data processing into two distinct pathways: a pre-processing pipeline that extracts and stores specified data items in structured formats, and a raw data storage pathway that preserves minimally processed data. This segmentation allows analysts to query structured extracted data for common analyses while accessing raw data for comprehensive searches, thereby reducing search complexity while maintaining analysis flexibility.
Solution Approach 2:
The patent introduces an intermediary search system that mediates between diverse data sources and analyst queries. This search system provides a unified interface that automatically routes queries to appropriate data pathways (structured or raw), translating complex multi-source search requirements into simplified analyst queries. The intermediary layer abstracts the underlying data complexity while preserving access to all raw data.
2Adaptability or versatility
If tools search data systems separately and collect results over a network, then data retrieval capability improves, but analysis efficiency and time consumption worsen
Solution Approach 1:
The patent merges multiple separate data system searches into a single unified search operation. The search system simultaneously queries across structured extracted data and unstructured raw data from multiple sources, combining results in one coordinated operation rather than requiring separate network calls to each data system. This merging dramatically improves analysis efficiency while maintaining the ability to retrieve data from all sources.
3Productivity
If data systems discard remainder of generated data during pre-processing, then data processing efficiency improves, but data analysis completeness deteriorates
Solution Approach 1:
The system performs preliminary extraction of specified data items during the pre-processing phase and stores them in structured formats for efficient querying. However, it also performs the preliminary action of preserving the remainder of generated data in raw format without discarding it. This dual preliminary action ensures both processing efficiency for common queries and complete data availability for comprehensive analyses.
Solution Approach 2:
The patent implements a selective discarding and recovering strategy: specified data items are extracted and stored in structured formats (effective use), while the remainder of generated data is preserved in raw format rather than discarded. This approach recovers the full value of all generated data for future analysis needs while maintaining efficient processing for commonly queried extracted items.
Data Source
AI summary
Systems and methods are described for unified processing of indexed and streaming data. A system enables users to query indexed data or specify processing pipelines to be applied to streaming data. In some instances, a user may specify a query intended to be run against indexed data, but may specify criteria that includes not-yet-indexed data (e.g., a future time frame). The system may convert the query into a data processing pipeline applied to not-yet-indexed data, thus increasing the efficiency of the system. Similarly, in some instances, a user may specify a data processing pipeline to be applied to a data stream, but specify criteria including data items outside the data stream. For example, a user may wish to apply the pipeline retroactively, to data items that have already exited the data stream. The system can convert the pipeline into a query against indexed data to satisfy the users processing requirements.