Message Batch Partitioning for Sequence Preservation
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
In parallel processing environments, ensuring the preservation of message sequence is challenging, leading to potential execution errors due to the asynchronous nature of messaging systems, where related requests may be transposed during processing.
Innovation Solution
Assigning each message of a batch to a partition of a log associated with a processing engine, storing offset values, and determining message batch completion by comparing target and stored offset values, allowing the distributor to ascertain when all messages have been processed.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Productivity
If messages are processed in parallel using multiple processors, then processing speed and productivity are improved, but message sequence preservation deteriorates causing execution errors
Solution Approach 1:
The patent divides the log into multiple partitions, with each partition assigned to a specific processing engine. Messages are routed to partitions based on their sequence requirements, allowing parallel processing while maintaining order within each partition. This segmentation enables multiple processors to work simultaneously without interfering with message sequence preservation.
2Reliability
If messages are processed sequentially to preserve sequence, then message sequence preservation is improved, but processing speed and productivity deteriorate
Solution Approach 1:
By segmenting the log into partitions and assigning each partition to a dedicated processing engine, the system achieves parallel processing while maintaining sequence preservation within each partition. Multiple processing engines can operate simultaneously on different partitions, improving overall productivity without sacrificing reliability.
Solution Approach 2:
The patent introduces a new dimension of organization by dividing the log into multiple partitions along the storage dimension. This allows the system to process messages in parallel across different partitions while maintaining sequential processing within each partition, effectively adding a spatial dimension to overcome the sequential processing limitation.
3Measurement precision
If offset values are tracked for each partition to determine batch completion, then accuracy of completion detection is improved, but system complexity increases
Solution Approach 1:
The system implements feedback by continuously tracking offset values for each partition and comparing them against target offset values to determine when a message batch is fully processed. This feedback mechanism provides accurate completion detection, allowing the system to know precisely when all messages in a batch have been processed across all partitions.
Solution Approach 2:
The target offset values are predetermined based on the message batch characteristics before processing begins. This preliminary action allows the system to have a clear completion criterion established in advance, simplifying the completion detection process while maintaining high accuracy.
Data Source
AI summary
A method and associated system. For each message of a message batch, the message is assigned by a consumer to a partition of a log. Each partition of the log is associated with a respective processing engine. The message batch includes messages that includes the message. The consumer stores an offset value for each partition. Responsive to completion, by a distributor, of sending a last message in the message batch to the consumer, the distributor ascertains a target offset value for each partition based on a current offset value. In response to a processing engine completing processing of a message assigned to the partition associated with the processing engine, the consumer updates the stored offset value associated with the partition. The distributor determines when all messages of the batch have been processed based on the target offset values and the stored offset values.