Event Messaging System for High Volume Network Data Processing
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Current systems for real-time complex event processing (CEP) face challenges in handling high volumes of events per second, requiring scalable solutions that can dynamically adjust resources and maintain performance across multiple nodes and datacenters, while also managing event messages efficiently over networks.
Innovation Solution
The implementation of a messaging system that utilizes a publisher-subscriber model, deploying multiple CEP engines in a cluster, distributing workload across devices, and using consistent hashing for efficient message routing and load balancing, allowing for self-healing and elastic scaling, and batching and compressing messages to optimize network usage.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Productivity
If current systems for real-time complex event processing are used, then processing capability is limited, but handling high volumes of events per second requires scalable solutions
Solution Approach 1:
The system segments the event processing workload by distributing events across multiple consumer devices based on device identifiers. Each consumer device processes a specific portion of the event stream, enabling parallel processing and scaling to handle high volumes of events per second without overwhelming a single system
Solution Approach 2:
The patent introduces a new dimension for system scaling by deploying consumer devices across multiple datacenters rather than simply increasing capacity within a single datacenter. This geographical distribution enables the system to handle larger event volumes while maintaining fault tolerance and scalability
2Productivity
If multiple CEP engines are deployed in a cluster, then workload distribution improves, but message routing complexity increases
Solution Approach 1:
The patent introduces an intermediary mechanism that maps device identifiers to consumer devices using consistent hashing. This intermediary layer simplifies message routing by providing a deterministic method to route events to the appropriate consumer device based on the event's device identifier, avoiding complex routing logic while maintaining efficient workload distribution
3Productivity
If network bandwidth is increased for high-volume event streams, then data transmission capacity improves, but network cost and energy consumption increase
Solution Approach 1:
The system segments the event stream into multiple parallel flows that are processed by different consumer devices. This segmentation enables efficient utilization of available network bandwidth by distributing traffic across multiple connections, achieving high data transmission capacity without requiring a single high-bandwidth connection that would consume excessive energy
Data Source
AI summary
Disclosed are a system comprising a computer-readable storage medium storing at least one program, and a computer-implemented method for event messaging over a network. An identification module receives first data identifying consumer devices available to receive event messages linked to a topic. The identification module receives second data identifying a producer device available to provide event messages linked to the topic. A provisioning module links a plurality of values to respective consumer devices that are linked to the topic. A scheduler module accesses a first data message linked to the topic. The first data message includes a key value. The scheduler module provides the first data message to a selected one of the consumer devices based on a comparison of the key value and the plurality of values of the respective consumer devices.