Sessionized Event Messaging for High-Volume Network Data
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing systems face challenges in efficiently processing high volumes of network data, particularly in complex event processing (CEP), as a single CEP engine lacks the necessary resources and scalability to handle large volumes of events at high velocity, variety, and variety, leading to performance issues and network instability.
Innovation Solution
Deploying multiple CEP engines in a cluster across devices, distributing workload, and using a messaging system that batches and compresses messages, allowing for elastic scaling and self-healing in case of failures or imbalances, with a publication-subscription model for producer and consumer devices.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Productivity
If a single CEP engine is used to process network data, then device complexity is reduced, but productivity and processing capacity are insufficient for high-volume data
Solution Approach 1:
The patent divides the CEP processing system into multiple independent CEP engines distributed across different devices. Each engine handles a portion of the event stream, enabling parallel processing and scaling to accommodate high-volume data while maintaining manageable complexity through modular architecture
Solution Approach 2:
The patent transitions from a single-device processing model to a multi-device distributed architecture, adding the spatial dimension of device distribution. This allows the system to scale processing capacity by adding more devices without proportionally increasing complexity at any single node
2Productivity
If multiple CEP engines are deployed in a cluster, then productivity and scalability are improved, but device complexity and system management difficulty increase
Solution Approach 1:
The patent implements a standardized message interface that enables CEP engines to communicate through a common protocol regardless of their specific implementation or location. This universal interface simplifies cluster management by providing consistent interaction patterns across diverse components
Solution Approach 2:
The patent introduces a message queue or event stream as an intermediary between CEP engines, which handles the complexity of coordination, load distribution, and fault tolerance. This mediator layer absorbs management complexity while providing simple publish-subscribe interfaces to individual engines
3Productivity
If data is transmitted without batching and compression, then processing speed is maintained, but network bandwidth consumption and data transmission time increase
Solution Approach 1:
The patent applies batching and compression to event data before transmission between CEP engines. This preliminary processing reduces the volume of data that needs to be transmitted over the network, decreasing transmission time and bandwidth consumption while maintaining the ability to process events in near real-time
Solution Approach 2:
The patent changes the physical parameters of data transmission by applying compression algorithms that reduce data size and batching that consolidates multiple events. These parameter changes optimize the trade-off between transmission efficiency and processing speed by adjusting data representation rather than fundamental processing architecture
Data Source
AI summary
Disclosed are a system comprising a computer-readable storage medium storing at least one program, and a computer-implemented method for event messaging over a network. A subscription interface receives data indicative of a subscription request for sessionized data. An allocation module allocates a sessionizer bank linked to the subscription request. A messaging interface module provisions identifiers linked to the respective processing engines of the sessionizer bank. The messaging interface module registers the allocated sessionizer bank as available to process event messages matching the subscription request by providing the provisioned identifiers. The messaging interface module receives event messages from a producer device linked by a collection server to a selected one of the processing engines of the sessionizer bank. The selected one of the processing engine processes the received event messages in accordance with session rule data linked to the subscription request to generate sessionized data.