Base Station Load Balancing With Multi-Teacher Reinforcement Learning
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing reinforcement learning algorithms face challenges in real-world applications due to poor data efficiency and inflexibility, requiring numerous interactions with the environment, and model-based methods suffer from model-bias issues, leading to unstable policy learning.
Innovation Solution
A multi-teacher reinforcement learning approach using multiple teacher networks to train student models through knowledge distillation, followed by ensemble methods to combine student models, addressing model-bias and improving learning efficiency.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If traditional reinforcement learning algorithms are used for traffic load balancing, then the system can learn control policies through environmental interactions, but the number of interactions required is excessively large, reducing data efficiency
Solution Approach 1:
The patent applies preliminary action by training teacher models offline using simulated environmental interactions before deploying to real-world applications. The teacher models pre-learn optimal control policies and transfer this knowledge to student models, eliminating the need for extensive real-time environmental interactions and significantly reducing training time while maintaining policy learning reliability
Solution Approach 2:
The patent uses copying by creating student models that replicate the functionality of teacher models through knowledge distillation. The student models copy the learned policies and decision-making frameworks from teacher models, allowing them to perform load balancing tasks without requiring the same extensive environmental interactions that the teacher models underwent during training
2Loss of time
If model-based reinforcement learning methods are used to reduce environmental interactions, then the number of interactions is reduced, but model-bias problems arise leading to unstable policy learning
Solution Approach 1:
The patent applies segmentation by dividing the reinforcement learning system into multiple teacher models and student models. Each teacher model can focus on specific aspects or subsets of the environment, and their segmented knowledge is then distributed to student models. This segmentation allows for more comprehensive coverage of the state space and reduces model-bias problems by incorporating diverse perspectives from multiple teacher models, thereby stabilizing policy learning
Solution Approach 2:
The patent uses parameter changes by dynamically adjusting the knowledge distillation process and model architecture parameters to optimize the balance between reducing environmental interactions and maintaining policy learning stability. By changing parameters such as the number of teacher models, knowledge distillation strength, and model architecture configurations, the system adapts to different task requirements while minimizing model-bias and maximizing learning stability
3Productivity
If multiple teacher networks are used for knowledge distillation, then prediction accuracy and learning efficiency are improved, but the system complexity increases
Solution Approach 1:
The patent applies universality by designing teacher and student models with universal architectures that can be applied across different load balancing scenarios. The models are designed to handle multiple tasks and environments through shared layers and parameters, allowing the same model structure to serve multiple functions. This universality reduces the actual complexity increase despite using multiple teacher networks, as the models share common computational infrastructure and training frameworks
Data Source
Figure 1
Figure 2
Figure 3A~3C
AI summary
A server may be provided to obtain a load balancing artificial intelligence (AI) model for a plurality of base stations in a communication system. The server may obtain teacher models based on traffic data sets collected from the base stations, respectively; perform a policy rehearsal process including obtaining student models based on knowledge distillation from the teacher models, obtaining an ensemble student model by ensembling the student models, and obtaining a policy model by interacting with the ensemble student mode; provide the policy model to each of the base stations for a policy evaluation of the policy model; and based on a training continue signal being received from at least one of the base stations as a result of the policy evaluation, update the ensemble student model and the policy model by performing the policy rehearsal process on the student models.