Virtual Network Allocation Using Deep RL for High-Dimensional States
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing dynamic virtual network allocation methods are limited in handling high-dimensional network states, leading to low accuracy and inefficient use of physical resources due to the need for pre-learning with discrete, low-dimensional network states.
Innovation Solution
A control apparatus employing deep reinforcement learning dynamically allocates virtual networks by receiving network configuration and user demand information, selecting actions based on a learned policy, and calculating rewards to optimize virtual network allocation in physical networks, allowing for continuous and high-dimensional state handling.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Measurement precision
If reinforcement learning is used for dynamic virtual network allocation, then the system can learn optimal control methods in advance, but it is limited to discrete and low-dimensional network states, resulting in low allocation accuracy
Solution Approach 1:
The patent transforms the network state representation from discrete, low-dimensional parameters to continuous, high-dimensional parameters. Specifically, it uses continuous variables to represent network states (such as traffic load, resource utilization) and employs a deep neural network to process these continuous inputs, thereby enabling the system to handle complex, high-dimensional network states while maintaining allocation accuracy.
Solution Approach 2:
The patent replaces the traditional reinforcement learning mechanism with a deep reinforcement learning approach. Instead of using conventional RL algorithms that require discrete state spaces, the invention integrates a deep neural network to approximate the value function or policy, substituting the mechanical RL update process with a neural network-based learning process that can handle continuous inputs.
2Adaptability or versatility
If deep reinforcement learning is applied to handle continuous and high-dimensional network states, then the system can process complex network configurations, but the computational complexity increases
Solution Approach 1:
The patent performs pre-training of the deep neural network offline before actual network operation. During this preliminary phase, the network learns optimal policies from simulated or historical data. Once trained, the network can make real-time allocation decisions without requiring complex online computations, thereby reducing the computational burden during actual network operation.
Solution Approach 2:
The patent uses a deep neural network to create a computational model (copy) of the complex network state space and optimal policies. Instead of directly processing high-dimensional continuous states through complex algorithms, the system uses the trained neural network as a surrogate model that approximates the optimal allocation decisions, simplifying the real-time decision-making process.
Data Source
AI summary
A control apparatus that dynamically allocates a virtual network for providing a network service to a physical network through deep reinforcement learning, includes: observation means for receiving input of network configuration information regarding the physical network, network observation information indicating an amount of physical resource of the physical network, and user demand information indicating an amount of demand for the physical resource generated due to a communication action of a user terminal for the network service, and observing a state st at a time t; allocation means for selecting an action at to change allocation of the virtual network to the physical network in accordance with a policy π in the state st; reward calculation means for calculating a reward rt+1 based on a state st+1 to which the state st has transitioned due to the action at; and learning means for learning the policy π using the reward rt+1.