In-Order Message Processing with Reactive Microservices
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing systems face challenges in processing in-order messages efficiently, particularly with reactive programming where message order cannot be guaranteed due to asynchronous code, leading to potential data loss and complexity in transaction implementations.
Innovation Solution
The solution involves performing in-order messaging within a message broker using unique message keys, providing parallel data streams grouped to each message broker partition, maintaining in-order processing through downstream publishers, and using a terminal sync message to identify the end of in-order streaming.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Productivity
If messages are streamed as smaller files with reactive programming to avoid parsing large bulk files, then processing efficiency and throughput are improved, but message order cannot be guaranteed due to asynchronous code
Solution Approach 1:
The system segments the data stream into smaller message files for parallel processing while maintaining logical ordering through partition-based routing. Each partition processes messages independently in order, allowing overall parallelism without sacrificing individual message sequence integrity.
Solution Approach 2:
The message broker acts as an intermediary that receives segmented messages and routes them to appropriate partitions. The partition key function serves as a mediator that ensures messages with the same key are routed to the same partition, preserving their relative order while enabling parallel processing across different partitions.
2Reliability
If complex transaction implementations are used with message brokers to ensure in-order processing, then message order is maintained, but throughput is impacted since all messages must be received and committed before a transaction can be recorded
Solution Approach 1:
The system segments the transaction processing by partition, where each partition maintains its own in-order processing independently. This eliminates the need for a single global transaction that would block all messages, allowing parallel transaction processing across partitions while maintaining order within each partition.
Solution Approach 2:
The system applies in-order processing partially at the partition level rather than excessively at the global level. Each partition processes messages in order independently, which is sufficient for most use cases and avoids the overhead of coordinating global transactions across all partitions.
3Productivity
If parallel data streams are provided for each message broker partition, then throughput is improved, but maintaining strict order across all streams becomes more complex
Solution Approach 1:
The system applies local quality by ensuring strict ordering within each partition's data stream while allowing different partitions to process in parallel. The ordering guarantee is applied locally at the partition level rather than globally across all partitions, reducing complexity while maintaining throughput.
Solution Approach 2:
The system resolves the ordering complexity by introducing a new dimension - the partition key. Messages are ordered within each partition dimension independently, transforming the single-dimensional global ordering problem into multiple independent one-dimensional ordering problems that are easier to manage.
Data Source
AI summary
A system, method, and computer-readable medium for processing in-order messages. A message broker is provided with unique message keys to performing in-order ensuring that a batch of messages are sent on the same message broker partition or data flow. A reactive generic data streaming processor provides a parallel data stream uniquely grouped to each message broker partition or data flow. In-order processing is maintained on each parallel data stream by using downstream publishers that preserve strict order in each data stream. A terminal sync message with a unique message key to identifies end of in-order streaming a message group.