Graph Data Structure Transformations for Event Journey Analysis
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Current data intake and query systems face challenges in seamlessly searching and analyzing diverse data types from various data sources, as their capabilities are often limited to internal data stores, and processing flows are unidirectional, preventing data from being routed to different destinations for comprehensive analysis.
Innovation Solution
A data intake and query system that extends search and analytics capabilities by employing a search process master and query coordinators combined with a scalable network of distributed nodes, allowing data to be collected and processed from diverse data systems, and providing processed data as search results, thereby enabling analysis across multiple data sources including external systems.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Adaptability or versatility
If data is stored in a unified data lake format for flexibility and comprehensive analysis, then data accessibility and analysis flexibility improve, but query performance and processing speed deteriorate due to the need to parse and filter large volumes of unstructured data
Solution Approach 1:
The system performs preliminary actions by generating and executing candidate queries against the data lake before the user's actual query is fully processed. This pre-computation of potential results allows the system to rapidly retrieve pre-filtered data when the user's query matches or overlaps with candidate queries, significantly reducing query processing time while maintaining flexibility in analyzing unstructured data
Solution Approach 2:
The system generates multiple candidate queries that may exceed the actual needs of a single user query. By preparing more queries than strictly necessary and caching their results, the system ensures that when a user query arrives, the answer is likely already computed or can be quickly assembled from existing results, trading some computational overhead for substantial query speed improvements
2Device complexity
If traditional unidirectional data processing flows are used, then system simplicity is maintained, but the ability to route data to multiple destinations for comprehensive analysis is limited
Solution Approach 1:
The system segments the data processing architecture into independent, modular components: data sources, query coordinators, candidate query generators, result analyzers, and multiple data destinations. Each component performs a specific function and can operate independently, allowing data to be routed to multiple destinations simultaneously while maintaining overall system simplicity through clear separation of concerns
Solution Approach 2:
The query coordinator and result analyzer components serve multiple functions: they coordinate candidate query generation, execute queries against the data lake, analyze results from multiple sources, and route outputs to various destinations including visualizations, reports, and downstream systems. This multi-functionality extends data routing capabilities without proportionally increasing system complexity
3Loss of information
If graph transformations are applied to query results to enable subset analysis, then analytical insights and data subset identification improve, but computational overhead and processing time increase
Solution Approach 1:
The system applies graph transformations to generate multiple candidate data subsets that may be more than needed for a single analysis task. By pre-computing and caching these transformed subsets, the system reduces the processing time required when users need to analyze specific data subsets, as the transformations are already performed or can be quickly applied to pre-filtered results rather than the entire data lake
Data Source
AI summary
Systems and methods are disclosed for supporting transformations of a graph generated from a query to event data. The event data may be unstructured event data, from which instances of a journey can be identified that represent sequences of related events describing actions performed in a computing environment. When evaluating journey instances, it can be helpful to visualize the instances as a graph. Depending on the instances viewed, a user may desire different modifications to the graph. While such modifications can be made when initially building instances from the unstructured event data, this can limit reuse of the resulting instances (since the modification would also be present when evaluating other subsets). To address this, embodiments of the present disclosure enable graph modifications to be applied to subsets of journey instances after building those instances from unstructured event data, increasing reuse of instances built from a query against the unstructured data.