Pipelined Event Processing in Distributed Data Systems
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing data analysis systems require persistent storage before analysis, which becomes inefficient as the number of data sources increases, and they struggle with real-time analytics across disparate data sources, leading to scalability and integration challenges.
Innovation Solution
A distributed data processing system that uses in-memory data processors to analyze and report events in real-time before persistent storage, employing a continuous query engine for real-time data integration across various sources, and a new data model (WAction) for tracking related events for search, aggregation, and visualization.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Productivity
If data is migrated and stored persistently before analysis using traditional ETL modules, then data can be analyzed, but the system becomes inefficient and cannot handle real-time analytics across multiple data sources
Solution Approach 1:
The system performs preliminary actions by setting up continuous query engines and in-memory data processors before data arrives. The framework pre-configures data processing pipelines, event subscriptions, and computation logic in advance, enabling immediate real-time analysis without requiring data migration and persistent storage first. This eliminates the traditional ETL bottleneck by having the analytical infrastructure ready beforehand.
Solution Approach 2:
The patent introduces an intermediary layer consisting of in-memory data processors and continuous query engines that sit between data sources and persistent storage. This intermediary framework enables real-time data processing and analysis without requiring data to be fully migrated to persistent storage first, thus resolving the contradiction between analysis capability and storage time.
2Quantity of substance
If the number of data sources increases, then more data is available for analysis, but the system complexity and integration challenges increase
Solution Approach 1:
The framework employs universal, multi-functional components including event sources, event processors, and continuous query engines that can handle multiple data sources through a common interface. This standardized architecture allows the system to scale to numerous data sources without proportionally increasing complexity, as the same core components serve multiple functions across different data sources.
Solution Approach 2:
The system segments the data processing architecture into independent, modular components: event sources, event processors, continuous query engines, and result sinks. Each component operates independently and can be configured to handle specific data sources, allowing the system to scale horizontally by adding more segmented components rather than increasing overall system complexity.
3Ease of manufacture
If traditional ETL modules are used for data processing, then data can be transformed and loaded, but real-time event processing and analytics are not achieved
Solution Approach 1:
The patent replaces the mechanical, batch-oriented ETL process with an event-driven, continuous processing system. Instead of extracting, transforming, and loading data in discrete batches, the system continuously processes events as they occur using in-memory computation and real-time query engines, substituting the traditional mechanical ETL workflow with a more responsive event processing mechanism.
Solution Approach 2:
The framework implements continuous data processing where transformation and analysis occur continuously as data flows through the system, rather than in periodic batches. The continuous query engine maintains persistent computations that process events continuously, ensuring real-time analytics capability while maintaining data transformation functionality.
Data Source
AI summary
A method includes receiving first data from a first data source that continuously generates a first set of records with first common fields. The first data is filtered by selecting a first sub-set of fields of the first common fields to form first filtered data. Second data is received from a second data source that continuously generates a second set of records with second common fields. The second data is filtered by selecting a second sub-set of fields of the second common fields to form second filtered data, where the first filtered data and the second filtered data are generated simultaneously. Rules are applied to the first filtered data and the second filtered data in real-time to identify selected real-time events. The selected real-time events are reported prior to persistently storing the first filtered data, the second filtered data and the selected real-time events.