Cellular Scheduler Timing Offset for Delayed Reward Feedback
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Deep reinforcement learning in cellular networks faces challenges due to over-the-air feedback delays, causing mismatches and conflicts with the time sequence of scheduling procedures, which hinders effective training of the scheduler.
Innovation Solution
A scheduling method where schedulers obtain reward feedback in one time unit and send scheduling decisions in a subsequent time unit, allowing sufficient time for calculation and coding, and multiple schedulers cooperate to adjust scheduling parameters based on exchanged information, ensuring similar or same scheduling policies and maximizing rewards.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Loss of time
If the scheduler obtains reward feedback and sends scheduling decisions in the same time unit, then the scheduling response time is minimized, but the scheduler cannot complete calculation and coding in time, causing over-the-air feedback delay and time sequence mismatch
Solution Approach 1:
The patent applies preliminary action by having the scheduler send the scheduling decision in advance (in the nth time unit) before obtaining the reward feedback (which arrives in the (n+1)th time unit). This allows the scheduling decision to be prepared and transmitted ahead of time, resolving the time sequence mismatch while ensuring sufficient calculation and coding time is available.
2Reliability
If the scheduler waits to obtain reward feedback before sending scheduling decisions, then deep reinforcement learning training can be performed correctly, but the scheduling procedure mismatches and conflicts with the over-the-air time sequence
Solution Approach 1:
The patent uses preliminary action by sending the scheduling decision before receiving the reward feedback. The scheduler determines the scheduling decision based on previous reward feedback and sends it in the nth time unit, while the reward feedback for that decision arrives later in the (n+1)th time unit. This maintains training accuracy while improving scheduling efficiency.
Solution Approach 2:
The patent applies dynamics by introducing a flexible time offset parameter n (where n≥1) that allows the system to adapt the timing between sending scheduling decisions and receiving reward feedback. This dynamic adjustment enables the system to balance between training accuracy requirements and scheduling efficiency needs.
3Adaptability or versatility
If multiple schedulers are used to improve adaptability to dynamic environments, then scheduling efficiency and resource utilization are enhanced, but the system complexity increases
Solution Approach 1:
The patent applies segmentation by dividing the scheduling system into multiple independent schedulers (first scheduler, second scheduler, etc.), each capable of independently obtaining reward feedback and sending scheduling decisions. This segmentation allows the system to handle dynamic environments more effectively while keeping each individual scheduler relatively simple.
Solution Approach 2:
The patent uses universality by designing multiple schedulers that all follow the same operational pattern and can perform the same functions. Each scheduler can obtain reward feedback, determine scheduling decisions, and send decisions with the same time offset n, making the system versatile and adaptable to different scenarios without requiring complex specialized designs for each scheduler.
Data Source
AI summary
A scheduling method, a scheduling system, and a scheduling apparatus. After obtaining a reward feedback for a previous scheduling decision sent by a terminal device, a scheduler sends, in N time units after the reward feedback is received, a current scheduling decision calculated based on the reward feedback, where N>1 and N is an integer. Sufficient time is provided for the scheduler to calculate and code the scheduling decision, thereby effectively resolving a problem that a scheduling procedure mismatches and conflicts with an over-the-air time sequence in response to the scheduler not obtaining a reward of the previous scheduling decision in time.