Multi-Agent Deep Reinforcement Learning for Virtual Network Allocation
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
The dynamic allocation method for virtual networks, which uses deep reinforcement learning, faces increased calculation time and decreased allocation accuracy with the increase in the number of servers and virtual networks, leading to longer control cycles and reduced accuracy in network resource allocation.
Innovation Solution
A control device employing multi-agent deep reinforcement learning dynamically allocates virtual networks by selecting agents to execute actions based on network state observations, calculating rewards, and updating action value functions to optimize virtual network allocation, thereby reducing calculation time and improving accuracy.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Measurement precision
If deep reinforcement learning is used for dynamic allocation of virtual networks, then allocation accuracy can be improved, but calculation time increases with the number of servers and virtual networks
Solution Approach 1:
The patent divides the virtual network allocation problem into multiple independent sub-problems, each handled by a separate agent. Each agent manages a specific subset of virtual networks or servers, allowing parallel processing and reducing overall calculation time while maintaining allocation accuracy through specialized learning for each segment.
Solution Approach 2:
The patent implements selective allocation where agents only perform allocation actions when necessary, rather than continuously reallocating all virtual networks. This partial action approach reduces calculation time by avoiding unnecessary computations while maintaining accuracy by focusing learning resources on critical allocation decisions.
2Measurement precision
If deep reinforcement learning is used for dynamic allocation, then allocation accuracy improves, but control cycle time increases
Solution Approach 1:
By segmenting the allocation problem into multiple agents working in parallel, the patent reduces the overall control cycle time. Each agent independently processes its subset of virtual networks simultaneously, avoiding sequential processing delays while maintaining comprehensive allocation accuracy across the entire system.
Solution Approach 2:
The patent implements pre-learning mechanisms where agents learn allocation policies in advance during training phases. This preliminary action allows the system to make rapid allocation decisions during actual operation without performing complex real-time calculations, thus reducing control cycle time while preserving allocation accuracy.
3Quantity of substance
If the number of servers and virtual networks increases, then system capacity improves, but calculation time and control cycle time increase
Solution Approach 1:
The patent scales the system by adding more agents rather than increasing the complexity of a single agent. Each agent handles a manageable subset of the growing number of virtual networks and servers, allowing the system to increase capacity linearly without exponentially increasing calculation time. This segmentation enables efficient parallel processing across multiple agents.
Solution Approach 2:
The patent transitions from a single-agent sequential approach to a multi-agent parallel architecture, adding the dimension of parallelism. This dimensional change allows the system to handle increased capacity by distributing work across multiple processing units simultaneously, preventing calculation time from increasing proportionally with system size.
Data Source
AI summary
Provided is a control device that includes one or more processors configured to: select an agent gk that executes an action that represents allocation of a virtual network onto a physical network at a time point t from a plurality of agents; observe a state st at the time point t with an input of network configuration information of the physical network, network observation information, and user demand information; select and execute, on basis of an action value function Q representing an expected value of a sum total of a reward received in future, an action atk from possible actions for the agent gk; calculate a reward rt using of the action atk, the state st, and a state st+1 at a time point t+1; and update the action value function Q with use of the action ajk, the state sj, and the state sj+1, where 1≤j≤t is satisfied.