Wireless Network Reward Simulation for Safe Reinforcement Learning
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Reinforcement learning in wireless networks faces challenges in effectively training machine learning models due to the impact of exploration on performance and resource usage, particularly when predicted actions may negatively affect network quality or resource utilization.
Innovation Solution
A method is introduced to obtain simulated rewards for predicted actions that do not meet reliability criteria, allowing the machine learning model to be trained based on these simulated rewards, thereby mitigating the negative effects on network performance during exploration.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Adaptability or versatility
If the machine learning model performs exploration actions to improve training, then the model's learning capability is improved, but network performance and resource utilization deteriorate
Solution Approach 1:
The patent introduces a simulated environment as an intermediary between the machine learning model and the real wireless network. The simulated environment receives predicted actions from the model, evaluates them against reliability criteria, and generates simulated rewards without affecting actual network performance. This mediator allows the model to explore and learn while the real network remains protected from harmful exploration actions.
Solution Approach 2:
The patent creates a copy of the wireless network environment for simulation purposes. Instead of performing exploration actions in the real network, the system uses a simulated copy to evaluate actions and generate training rewards. This copying approach enables the model to learn from explored actions while the original network maintains stable performance.
2Adaptability or versatility
If the machine learning model explores suboptimal actions for training purposes, then the model's adaptability is improved, but resource utilization deteriorates
Solution Approach 1:
The simulated environment acts as an intermediary that absorbs the resource cost of exploration. By evaluating suboptimal actions in simulation rather than in the real network, the system generates training data without consuming actual network resources. The mediator environment handles the exploratory computations while the real network resources remain available for productive use.
3Reliability
If the predicted action is rejected due to reliability criteria, then network performance is protected, but the machine learning model cannot learn from that action
Solution Approach 1:
When a predicted action is rejected by reliability criteria in the real network, the system creates a simulated copy of that action in the simulated environment. The simulated environment evaluates the rejected action and generates a simulated reward, which is then used to train the machine learning model. This copying mechanism ensures that neither the real network nor the model learning is compromised.
Solution Approach 2:
The patent converts the harmful effect of rejected actions into a beneficial training opportunity. Instead of discarding actions that fail reliability checks, the system uses them as input to the simulated environment, which generates simulated rewards that improve model learning. The previously harmful rejected actions become valuable training data through this conversion process.
Data Source
Figure 1
Figure 2
Figure 3
AI summary
A method includes receiving, by a user device from a network node, a reliability criteria; providing, by the user device, one or more measurements or context information as inputs to a machine learning model; receiving, by the user device from the machine learning model, a predicted action based on the inputs; determining, by the user device, whether the reliability criteria is met for the predicted action; performing, by the user device, the predicted action if the predicted action meets the reliability criteria; and if the predicted action does not meet the reliability criteria: obtaining a simulated reward for the predicted action; and training or updating the machine learning model based on the simulated reward for the predicted action.