Preprocessing Node Dynamic Routing for Message Processing
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing streaming processing systems face challenges in efficiently processing messages according to specific rules, particularly in scenarios like network management where association analysis and message merging are required, and in ensuring sequential processing of messages in large-scale data environments.
Innovation Solution
A message processing method and system that dynamically calculates a processing path for messages based on rule instances, allowing preprocessing nodes to search for and establish paths to processing nodes where corresponding rule instances are located, enabling efficient distribution and processing of messages according to defined rules.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Productivity
If traditional network management processes messages using conventional methods, then the system maintains simplicity in architecture, but the message processing capability and efficiency are insufficient for large-scale data environments
Solution Approach 1:
The system segments message processing into distinct functional components: preprocessing nodes that perform initial filtering and routing, processing nodes that execute specific processing operations, and a management node that coordinates rule instances. This segmentation enables scalable message processing while maintaining architectural clarity through defined interfaces and responsibilities between components.
2Productivity
If streaming processing is used to improve message processing efficiency, then processing speed and throughput increase, but the ability to perform rule-based association analysis and sequential processing becomes difficult to implement
Solution Approach 1:
The system implements dynamic rule instance registration and routing where processing paths are determined at runtime based on message content and matching rules. The preprocessing node dynamically selects which processing node to forward messages to based on the rule instance that best matches the message characteristics, enabling flexible adaptation to different processing requirements while maintaining high-speed streaming processing.
Solution Approach 2:
The preprocessing node acts as an intermediary that receives messages from data sources, evaluates them against multiple rule instances, and routes them to appropriate processing nodes. This intermediary layer enables the system to maintain simple high-speed message forwarding while incorporating complex rule-based processing logic, as the preprocessing node mediates between the fast streaming infrastructure and the flexible rule evaluation mechanism.
3Quantity of substance
If multiple processing nodes are deployed to handle large volumes of messages, then processing capacity and distribution are improved, but the complexity of managing and coordinating processing paths increases
Solution Approach 1:
The system performs preliminary registration of processing nodes and their capabilities with the management node before actual message processing begins. Processing nodes pre-register the types of messages they can handle and the rule instances they support. This preliminary action enables the preprocessing node to quickly route messages to appropriate processing nodes without complex runtime discovery, reducing the complexity of managing distributed processing paths while maintaining high processing capacity.
Data Source
Figure 1~2
Figure 3~5
Figure 6
AI summary
A message processing method, apparatus and system: in response to receiving a first message of an event sent by a data source, a preprocessing node searching for a processing object among rule instances created by a system, the processing object comprising a rule instance of the first message; the preprocessing node establishing a processing path according to the rule instance, the processing path comprising addressing information of a processing node where the rule instance is located; and the preprocessing node sending the first message to the processing node according to the processing path so that the first message is processed by using the rule instance.