Master-Slave DRL Scheduling for Large Queue Management
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Deep reinforcement learning (DRL) algorithms face challenges in multi-objective task scheduling (MOTS) due to difficulties in defining a single reward function for integrating conflicting objectives and ensuring consistent processing of items regardless of their position in the queue, and they struggle with large queue sizes, requiring larger architectures and longer training times.
Innovation Solution
A master-slave DRL system is introduced, where a master DRL unit manages a queue and selects items to be processed by a slave DRL unit, allowing for efficient handling of larger queue sizes and ensuring consistent processing by dividing the queue into sub-queues and using a hierarchical structure to manage tasks.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Productivity
If a single DRL algorithm is used to manage the entire queue, then the system can handle the queue management task, but it requires larger neural network architectures and longer training times when dealing with large queue sizes
Solution Approach 1:
The queue management system is divided into multiple independent DRL algorithms, each responsible for a specific sub-queue. This segmentation allows each algorithm to process a smaller portion of the overall queue, reducing the computational burden and training requirements for each individual algorithm while maintaining the ability to handle large-scale queue management through coordinated operation of multiple algorithms
2Productivity
If a single reward function is used to integrate multiple conflicting objectives, then the system can optimize multiple goals, but it becomes difficult to define the reward function and ensure consistent processing of items regardless of their position in the queue
Solution Approach 1:
The reward function design is segmented by assigning different reward functions to different DRL algorithms based on their specific sub-queue responsibilities. Each algorithm optimizes its own local objectives with a tailored reward function, avoiding the complexity of defining a single comprehensive reward function that must balance all conflicting objectives simultaneously
Solution Approach 2:
Different parts of the queue (sub-queues) are assigned different optimization goals and reward functions appropriate to their specific characteristics. This allows each segment to be optimized for its local requirements while contributing to the overall system performance, ensuring consistent processing within each sub-queue context
3Quantity of substance
If DRL is applied to massive queues, then the system can process large volumes of tasks, but it results in large neural architectures and long training times
Solution Approach 1:
The massive queue is divided into multiple smaller sub-queues, each managed by a separate DRL algorithm. This segmentation enables parallel training of multiple smaller algorithms instead of training one large algorithm, significantly reducing the training time for each algorithm while collectively handling the same large queue size through coordinated operation
Data Source
AI summary
A master-slave scheduling system, comprising (a) a master DRL unit comprising: (i) a queue containing a plurality of item-representations; (ii) a master policy module configured to select a single item-representation from the queue and submit to the slave unit; (iii) a master DRL agent configured to (a) train the master policy module; and (b) receive an updated item-representation from the slave unit, and update the queue;(b) The slave DRL unit comprising: (i) a slave policy module receiving a single item-representation, selecting a single task entry and submitting to a slave environment for performance; (ii) a slave DRL agent configured to: (a) train the slave policy module; (b) receive an item-representation from the master DRL unit, and submit to the slave policy module; (c) receive an updated item-representation from the slave's environment, and submit the same to the master DRL unit; and (iii) the slave DRL agent.