Journey Instance Update in Unstructured Event Data
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Current data intake and query systems face challenges in seamlessly searching and analyzing diverse data types from various data sources, as their capabilities are often isolated to internal data stores, lacking the ability to route data to different destinations and extending search and analytics operations beyond their scope.
Innovation Solution
A data intake and query system that employs a search process master and query coordinators combined with a scalable network of distributed nodes to collect and process data from diverse data systems, enabling search and analytics operations across internal and external data sources, including MySQL, PostgreSQL, NoSQL data stores, cloud storage, and Hadoop distributed file systems.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Productivity
If data is pre-processed and only specified data items are extracted and stored, then retrieval and analysis efficiency is improved, but data flexibility and completeness are reduced
Solution Approach 1:
The system performs preliminary actions by creating indexes and extracting metadata during data ingestion, but maintains the option to retrieve and analyze additional data elements on-demand later, balancing pre-processing benefits with future flexibility
Solution Approach 2:
The system dynamically adjusts the level of data processing and retrieval based on query requirements, allowing analysts to access pre-processed data for routine queries while enabling on-demand extraction of additional data elements for ad-hoc analysis
2Adaptability or versatility
If massive quantities of raw data are stored for later retrieval, then data analysis flexibility is improved, but storage costs and data management complexity increase
Solution Approach 1:
The system segments data into different levels of processing (raw data, extracted fields, indexes) and stores them in appropriate formats and locations, allowing efficient management of large data volumes while maintaining analysis flexibility
Solution Approach 2:
The system introduces intermediate data structures such as extracted fields and metadata that serve as mediators between raw data and analysis queries, reducing the complexity of managing raw data while preserving analytical capabilities
3Loss of information
If search and analytics operations are extended to external data sources, then data insights comprehensiveness is improved, but system complexity and data integration challenges increase
Solution Approach 1:
The system implements universal data access capabilities that can search and analyze data from multiple sources (internal data lakes, external databases, cloud storage, Hadoop systems) through a unified interface, comprehensiveness while managing complexity through standardization
Data Source
AI summary
Systems and methods are disclosed for efficiently storing information identifying journey instances within unstructured event data of a data intake and processing system. Each journey instance is illustratively associated with a series of events within the unstructured event data occurring over a journey duration. Because the unstructured event data may be constantly updated, any given inspection of the event data may yield both complete and incomplete instances. Storage of instance data over time can require updating of prior incomplete journey instances with complete versions of such instance detected at a later point in time. However, a data store of the unstructured event data may be unsuited for such updating, as the store may maintain version information for deleted data to reduce possibility of data loss. To address this issue, a separate structured data store, such as a columnar time series data store, is provided to efficiently store instance information.