Message Queue Processing with Decoupled Pullers and Task Execution
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing data processing systems integrate message pulling and processing into a single step, preventing independent expansion based on resource requirements, leading to inefficiencies and limitations in managing concurrency and scalability.
Innovation Solution
A system comprising a control server, a puller, and a task execution manager that separates data pulling and processing, allowing for independent expansion and management of resources, including a scheduling strategy to select pullers and task execution managers, and adds data states to databases for task management.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Device complexity
If message pulling and processing are integrated into one step, then the system structure is simple, but independent expansion based on resource requirements cannot be performed
Solution Approach 1:
The patent divides the integrated message processing system into separate components: a message pulling module and a message processing module. The pulling module is responsible for acquiring messages from the message queue, while the processing module handles message execution. This segmentation allows each module to be independently scaled and managed based on specific resource requirements, resolving the contradiction between system simplicity and expansion capability.
2Ease of manufacture
If message pulling and processing are integrated, then implementation is straightforward, but concurrency management and scalability are limited
Solution Approach 1:
By separating the pulling and processing functions into distinct modules, the system can manage concurrency more effectively. The pulling module can operate independently to fetch messages, while the processing module can handle multiple message executions in parallel. This architectural division enables better resource utilization and scalability without significantly complicating the implementation.
Solution Approach 2:
The patent introduces a message queue as an intermediary component between the pulling module and the processing module. This mediator allows for decoupled communication, where the pulling module places messages in the queue and the processing module consumes them asynchronously. This intermediary mechanism enhances concurrency management and scalability while maintaining implementation simplicity.
Data Source
AI summary
The present disclosure provides a system for processing data. A specific implementation is as follows: a control server acquires, based on a preset scheduling strategy, a second puller from the first puller, and a second task execution manager from the first task execution manager; and controls the second puller and the second task execution manager to perform an operation; a second puller acquires data of a to-be-processed message in a message queue; serializes the data of the to-be-processed message to obtain to-be-stored data; stores the to-be-stored data into a database to obtain stored first data; and adds a data state of the first data to the database; and a second task execution manager acquires second data from the database, and executes a task corresponding to the second data; and updates the data state based on a task execution result of the second data.