Data Processing System Using ID-Based Sequential Routing
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
The existing data processing systems using a publish-subscribe model face increased user workload due to changes in the data processing sequence, requiring frequent adjustments in channel settings with the broker and processors, which complicates cooperative data processing among multiple processors.
Innovation Solution
A data processing system with multiple processors performing sequential subprocesses, where each processor determines based on identification information whether to process received data and assigns new identification information to processed data, simplifying data exchange and reducing the need for frequent channel setting changes.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Ease of operation
If multiple channels are set with a broker to process data in a predetermined order, then data distribution to processors in a predetermined order is enabled, but the workload of the user increases when the data processing sequence changes
Solution Approach 1:
The patent introduces a message queue as an intermediary component between the producer and consumers. The message queue manages the data processing sequence and channel settings, acting as a mediator that absorbs the complexity of sequence management. When the processing sequence changes, only the message queue configuration needs adjustment, not the broker settings or processor subscriptions, thereby reducing user workload while maintaining flexibility.
Solution Approach 2:
The system enables processors to automatically determine whether to process received data based on identification information (such as sequence numbers or tags) without requiring manual configuration changes. The processor independently evaluates if the data matches its processing criteria and acts accordingly, eliminating the need for users to manually adjust channel settings when processing sequences change.
2Adaptability or versatility
If channel settings are changed in the broker and processor settings when the data processing sequence changes, then the data processing order can be adjusted, but the complexity of configuration management increases
Solution Approach 1:
The message queue serves as a configuration management intermediary that centralizes the management of processing sequences. Instead of distributing configuration changes across multiple brokers and processors, all sequence logic is managed in the message queue, simplifying configuration management while maintaining adaptability.
Solution Approach 2:
The patent segments the configuration management function from the data exchange function. The message queue handles sequence configuration independently, while the broker handles data routing. This segmentation allows configuration changes to be made in isolation without affecting the overall system architecture, reducing configuration management complexity.
3Adaptability or versatility
If a publish-subscribe model is used for data exchange between processors, then data distribution flexibility is improved, but the ability to ensure sequential processing order becomes difficult to maintain
Solution Approach 1:
The message queue performs preliminary actions by pre-tagging messages with identification information that indicates their processing sequence and destination. Before data is distributed to processors, the message queue has already organized the data in the correct sequence and attached appropriate metadata, ensuring that sequential processing is maintained even in the flexible publish-subscribe model.
Solution Approach 2:
The system uses feedback mechanisms where processors report back to the message queue about their processing status and the identification information of received messages. This feedback loop allows the message queue to track the sequential processing state and ensure that data is distributed in the correct order, maintaining reliability while preserving publish-subscribe flexibility.
Data Source
AI summary
A data processing apparatus (10) includes multiple data processors (300) that each perform a set subprocess of multiple sequential subprocesses included in a process sequence and a data distributor (210) that mediates data exchange between the multiple data processors (300) in accordance with a publish-subscribe model. An ID checker (330) determines, based on identification information assigned to received data received from the data distributor (210), whether received data is to be processed in the subprocess set to be performed by the data processor (300). A data processing executor (340) performs the set subprocess when the ID checker (330) determines that the received data is to be processed in the set subprocess. A data converter (310) assigns new identification information to data resulting from the set subprocess performed by the data processing executor (340) and transmits the data assigned with the new identification information to the data distributor (210).