Journey Model for Collapsing Data Steps in Query Systems
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Current data intake and query systems face challenges in seamlessly searching and analyzing diverse data types across various data sources, as their capabilities are often isolated to internal data stores, lacking the ability to route data to different destinations and extending search and analytics operations beyond their scope.
Innovation Solution
A data intake and query system that employs a search process master and query coordinators combined with a scalable network of distributed nodes to collect and process data from diverse data systems, enabling search and analytics operations across internal and external data sources, including enterprise systems and open-source technologies, through big data analytics.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Adaptability or versatility
If data is stored in internal data stores only, then search and analytics operations are contained within system scope, but the system cannot extend search capabilities to external data systems
Solution Approach 1:
The patent introduces a journey model as an intermediary structure that mediates between internal data stores and external data systems. This journey model acts as a bridge, enabling the search process master to route queries through multiple data sources without requiring direct integration of each external system into the core architecture, thus extending search capabilities while managing complexity through a standardized intermediary layer
2Adaptability or versatility
If data is collected from multiple diverse data sources, then comprehensive data analysis is enabled, but data processing and routing become complex
Solution Approach 1:
The patent segments the data processing function by introducing separate components: a search process master that handles query coordination, journey models that define data collection paths, and individual data sources that provide specific data types. This segmentation allows each component to specialize in handling specific data sources or query types, reducing overall processing complexity while maintaining the ability to handle diverse data from multiple sources
Solution Approach 2:
The journey model serves as a universal framework that can accommodate multiple diverse data sources through a single standardized interface. Rather than requiring separate processing logic for each data source type, the system uses the universal journey model to route and process data from various sources (internal databases, external systems, cloud services) through consistent processing steps, thereby enabling data source diversity without proportional increase in processing complexity
3Adaptability or versatility
If raw data is stored for later analysis, then data flexibility is improved, but storage requirements and data management complexity increase
Solution Approach 1:
The patent applies preliminary action by pre-defining journey models that specify how data should be collected, routed, and processed before actual analysis occurs. These journey models are created in advance with predefined parameters and routing logic, so when analysis is needed, the system can simply instantiate the appropriate journey model without needing to manage complex ad-hoc data processing logic, thus reducing data management complexity while maintaining analysis flexibility
Data Source
AI summary
A system can collapse steps into an aggregate step to simplify analysis while maintaining underlying data that forms each of the steps collapsed into the aggregate step. The steps may or may not be related in a sequence or grouping of steps. The aggregate step may be a new step that comprises the data of the individual steps used to form the aggregate step. Alternatively, the aggregate step may be a virtual step that may reference or link to the steps used to form the aggregate step, but may not include the data itself. By forming aggregate steps, filtering and notification generation can be simplified. Further, extraneous data can be collapsed into a single aggregate step, which can be particularly advantageously when analyzing large data sets.