Consistent Hashing for Message Sequencing in Scale-Out Workers
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
In scale-out architectures using the worker queue pattern, maintaining the exact sequence of messages related to the same resource is challenging due to parallel asynchronous processing, which can lead to inconsistent processing states if messages are processed out of order.
Innovation Solution
The implementation of consistent-hashing ensures proper sequencing by distributing messages to specific queues based on hash results, guaranteeing that messages for the same resource are processed in the correct order by always being sent to the same worker.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Productivity
If parallel asynchronous processing is used to increase throughput, then productivity is improved, but message sequencing reliability deteriorates
Solution Approach 1:
The system segments the message stream into multiple independent queues, where each queue is assigned to a specific worker node. Messages for the same resource are guaranteed to be in the same queue, preserving their sequential order while allowing different queues to be processed in parallel, thus resolving the contradiction between throughput and sequencing reliability.
Solution Approach 2:
A consistent hashing mechanism is introduced as an intermediary between the message stream and worker nodes. This intermediary distributes messages to appropriate queues based on resource identifiers, ensuring that messages for the same resource always route to the same queue while enabling parallel processing across multiple workers.
2Productivity
If multiple workers process messages in parallel, then productivity is improved, but processing consistency deteriorates
Solution Approach 1:
By segmenting the workload into resource-specific queues assigned to individual workers, the system ensures that each worker maintains a consistent processing state for its assigned resources. This segmentation prevents interleaved processing of related messages that would cause state inconsistency, while still allowing parallel execution across workers.
Solution Approach 2:
Each worker node is assigned specific resource queues based on consistent hashing, giving it specialized responsibility for those resources. This local quality assignment ensures that each worker maintains consistent processing state for its designated resources, while the overall system achieves parallel processing throughput.
3Productivity
If messages are distributed to different workers, then productivity is improved, but sequencing accuracy deteriorates
Solution Approach 1:
The system segments messages into resource-specific queues, ensuring that all messages for a given resource are confined to a single queue assigned to one worker. This segmentation guarantees sequencing accuracy for each resource while enabling parallel processing across different resource queues, resolving the contradiction between throughput and order preservation.
Solution Approach 2:
The consistent hashing mechanism acts as an intermediary that deterministically routes messages to specific queues based on resource identifiers. This intermediary ensures that messages for the same resource always follow the same path to the same queue, preserving their sequential order while enabling parallel processing of different resource streams.
Data Source
AI summary
Example embodiments of the present invention provide a method and a system for using consistent-hashing to ensure proper sequencing of message processing in a scale-out environment. The method includes receiving a stream of a plurality of messages and performing a consistent-hash on each message of the plurality of messages. Each message then may be distributed toward a respective downstream worker of a plurality of downstream workers for processing according to results of the hash.