Event Consumer State Management for Aggregated Event Platforms
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
In event consumer management systems, efficiently handling high volumes of events across multiple initial topics and partitions within an aggregated event platform is challenging due to the need for event consumers to skip irrelevant events and manage idle states, leading to increased system load and overhead.
Innovation Solution
Implementing a method where event consumers manage their states based on initial topics within a batch of events received on an aggregate topic, allowing for independent consumption and processing in batches, rather than one by one, and dynamically updating their execution status to optimize resource usage.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Use of energy by moving object
If event consumers are put to sleep (idle state) to reduce system load, then energy consumption is reduced, but processing time increases when events need to be consumed
Solution Approach 1:
The event consumer instances dynamically transition between idle and running states based on real-time event subscription requirements. The system monitors subscription status and automatically wakes consumers from idle state when events are published, ensuring energy efficiency during low-activity periods while maintaining rapid responsiveness when events occur.
2Speed
If event consumers are always running to ensure immediate event consumption, then processing speed is maintained, but system load increases
Solution Approach 1:
Event consumers operate in periodic cycles, transitioning between idle and active states. They remain in idle state during periods without events, reducing system load, and are automatically activated when events are published to the aggregate topic, ensuring processing speed is maintained only when necessary.
3Reliability
If event consumers process all events in aggregate topics, then complete event data is obtained, but filtering overhead increases
Solution Approach 1:
The system extracts and processes only the specific events relevant to each event consumer's initial topic from the aggregate topic stream. Event consumers subscribe to aggregate topics but filter and process only events matching their specific initial topic, eliminating the need to skip through irrelevant events and reducing filtering overhead while maintaining data completeness.
Data Source
AI summary
A method and system for event consumers management are described. A batch of events that is stored in an event recordation system according to an aggregate topic is received based on the aggregate topic. Each event from the batch of events is associated with an initial topic to be used by a respective instance of an event consumer from a set of event consumers to consume events. A first initial topic associated with one or more events from the batch of events is determined. Based on the first initial topic, a state of a first event consumer is updated to a running state. The updated state of the first event consumer causes execution of a first instance of the first event consumer on a server of a cluster of servers for consuming events from the event recordation system based on the first initial topic.