Message Processing Middleware Grouping Nodes
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing web services systems using a publish-subscribe model face challenges with message processing, particularly when multiple subscribers receive the same messages, as single-node systems become overwhelmed, leading to interruptions and inability to handle large message traffic, resulting in hardware upgrades and potential message loss.
Innovation Solution
A message processing system where message subscribers are grouped and each group has multiple nodes sharing the workload, with a middleware that distributes messages based on group IDs, ensuring fault-tolerant operation and efficient processing of large message volumes by dynamically adding or removing nodes and using long connections for real-time message transmission.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If a single node is used for message subscriber, then device complexity is reduced, but reliability deteriorates because message processing is interrupted when the node fails
Solution Approach 1:
The message subscriber is divided into multiple nodes (first node, second node, etc.) that work in parallel. Each node can independently process messages, so when one node fails, other nodes continue processing, ensuring message processing continuity without requiring a complex centralized structure
2Productivity
If a single node processes all messages, then device complexity is low, but productivity deteriorates when large message traffic needs to be handled
Solution Approach 1:
Multiple subscriber nodes are deployed to handle different portions of message traffic simultaneously. The middleware distributes messages to appropriate nodes based on subscription information, enabling the system to scale processing capacity by adding nodes without fundamentally changing the subscriber configuration model
3Productivity
If hardware is upgraded to handle large message traffic, then productivity improves, but loss of substance increases due to constant hardware upgrades
Solution Approach 1:
The system dynamically adjusts message processing distribution among nodes based on current message traffic volume and node availability. When traffic increases, more nodes can be activated or existing nodes handle increased load without requiring immediate hardware upgrades, as the system adapts its resource utilization in real-time
4Adaptability or versatility
If multiple subscribers subscribe to the same topic, then adaptability improves, but device complexity increases as middleware must transmit to multiple subscribers
Solution Approach 1:
The middleware maintains separate subscription information for each subscriber node independently. When a message is published, the middleware segments the transmission task by routing to specific nodes based on their subscriptions, avoiding the need for complex multi-to-many routing logic while preserving flexible subscription capabilities
Data Source
Figure 1
Figure 2
Figure 3
AI summary
Processing a message includes receiving at a message processing system the message that is distributed by a message distributor, the message having a message type, based at least in part on the message type, locating among a plurality of subscriber groups a subscriber group that subscribes to the message type, each of the plurality of subscriber groups having a unique identifier and comprising one or more message subscriber nodes, at least some of the one or more message subscriber nodes in each of the plurality of subscriber groups being connected to the message processing system, and transmitting the message to a message subscriber node that belongs to the subscriber group that is located.