Distributed Event Processing with Multi-Thread Commit Coordination
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Conventional management frameworks for event message processing in data streaming platforms often result in high infrastructure costs and increased risk of event message loss due to single consumer thread per topic partition, especially when asynchronous publishing is used for performance improvement.
Innovation Solution
A management framework utilizing a multiple-consumers/multiple-producers design, where data is processed through a consumer thread, loaded onto a blocking queue, processed by auxiliary consumer threads, published to a producer buffer with a load balancer, and recorded in a shared collection, ensuring efficient event message processing with low latency and no loss.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Productivity
If a single consumer thread per topic partition is used, then infrastructure costs are reduced, but event message throughput is limited and risk of message loss increases
Solution Approach 1:
The system segments the consumer side into multiple consumer threads, each capable of independently processing event messages from topic partitions. This segmentation enables parallel processing across multiple threads while maintaining individual thread reliability through coordinated commitment protocols, thereby increasing overall throughput without sacrificing message delivery guarantees.
Solution Approach 2:
The patent introduces an intermediary coordination mechanism that manages the interaction between multiple consumer threads and the commit protocol. This intermediary layer ensures that despite multiple threads accessing data simultaneously, the system maintains strong consistency and reliability by coordinating commit operations, thus preventing message loss while enabling parallel processing.
2Productivity
If asynchronous publishing is used on the consumer thread, then performance is improved, but risk of event message loss is significantly increased
Solution Approach 1:
The system implements beforehand cushioning by maintaining buffered copies of event messages and using coordinated commit protocols that ensure messages are not lost even if asynchronous operations fail. The commit protocol acts as a safety mechanism that verifies message delivery before confirming success, cushioning against potential failures in the asynchronous publishing path.
Solution Approach 2:
The patent implements feedback mechanisms where the system continuously monitors the status of asynchronous publishing operations and adjusts its behavior accordingly. The commit protocol provides feedback loops that verify message delivery status, allowing the system to detect and recover from failures, thus maintaining reliability despite using asynchronous publishing for performance improvement.
3Productivity
If multiple partitions are added to support higher consumption throughput, then throughput is improved, but infrastructure costs increase
Solution Approach 1:
Instead of adding more partitions to increase throughput, the system segments the processing capacity by introducing multiple consumer threads that can share existing partitions. This approach increases consumption throughput by utilizing better resource utilization of existing infrastructure rather than expanding infrastructure, thereby avoiding increased costs while achieving higher throughput.
Solution Approach 2:
The patent introduces dynamic consumer thread allocation and coordination mechanisms that allow the system to adaptively utilize existing partition resources more efficiently. By dynamically managing multiple consumer threads accessing the same partitions with proper coordination, the system achieves higher throughput without the need to statically add more partitions, thus avoiding associated infrastructure cost increases.
Data Source
AI summary
A method for providing a management framework for distributed event processing is disclosed. The method includes retrieving data from a distributed event streaming platform based on a data fetch size configuration, the data including event messages; submitting a batch of the data to a consumer thread based on a data polling configuration; loading the batch of the data onto a data structure that enables coordination of processing threads, the data structure including a blocking queue; processing each of the event messages in the batch of the data from the data structure based on predefined logics; publishing each of the processed event messages in the batch of the data to a producer buffer based on a selection logic, the producer buffer including a load balancer; and recording an agreement for each of the published event messages in the batch of the data to a shared collection.