Subnetwork RL Agents for Topology-Independent Network Actions
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Current software products fail to provide effective guidance on network actions due to the difficulty in codifying domain expertise into explicit rules, especially in complex, multi-vendor, or multi-domain scenarios, and require significant time and resources from professional services.
Innovation Solution
A process involving the creation of a global Reinforcement Learning (RL) model from subnetwork RL agents, trained on end-to-end metrics independent of specific topology, to recommend network actions, which includes training, testing, and applying the model to an Action Recommendation Engine (ARE) for network optimization.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If expert rules are used to provide network action guidance, then reliability is improved for simple cases, but device complexity increases and adaptability deteriorates for complex multi-vendor scenarios
Solution Approach 1:
The patent segments the network into multiple subnetworks, each managed by independent RL agents. This allows the system to handle complex multi-vendor scenarios by dividing the overall network management task into smaller, manageable sub-tasks that can be solved independently and then coordinated through the global model.
Solution Approach 2:
The global RL model acts as an intermediary that coordinates between multiple subnetwork RL agents. It learns to make decisions based on high-level network state observations without needing to understand the complex internal details of each subnetwork, thereby adapting to multi-vendor scenarios without requiring extensive domain expertise codification.
2Measurement precision
If explicit network state determination is performed, then measurement precision is improved, but loss of time increases due to the difficulty and expense of state determination
Solution Approach 1:
The patent implements partial state determination by having RL agents observe only the metrics necessary for their specific subnetwork tasks rather than determining the complete network state. This partial observation approach reduces the time and resources required while still enabling effective decision-making for each agent's scope.
Solution Approach 2:
Each RL agent independently determines its own relevant state metrics for its subnetwork without requiring centralized state determination. This self-service approach allows parallel state determination across multiple agents, significantly reducing overall time while maintaining necessary measurement precision for each agent's decisions.
3Reliability
If professional services are used to provide network action guidance, then reliability is improved, but loss of time and resources increases significantly
Solution Approach 1:
The system implements self-service through automated RL agents that independently learn and execute network optimization decisions without requiring professional services intervention. The agents continuously learn from network data and automatically adapt to changing conditions, eliminating the need for time-consuming professional services while maintaining reliable action guidance.
Solution Approach 2:
The RL agents incorporate feedback mechanisms where they continuously learn from the outcomes of their actions and adjust their policies accordingly. This feedback-driven learning enables the system to improve its reliability over time without requiring external professional services, as the agents autonomously refine their decision-making based on observed results.
Data Source
AI summary
A method for optimizing network performance using reinforcement learning (RL) agents is disclosed. The method includes identifying multiple network segments within a network, each including network nodes; generating and training respective RL agents for at least a subset of these segments based on performance metrics indicative of data flow within each segment, independently of specific segment topology information; receiving outputs from the trained RL agents, including policies or performance evaluations; generating recommendations based on the received outputs; and causing network actions to be implemented based on these recommendations. In various embodiments, the RL agents utilize metrics such as Quality of Service (QOS), Quality of Experience (QoE), or radio resource management parameters. Recommended actions may include switching traffic paths, adjusting wireless parameters, and proactively preventing network congestion to enhance network operation and user experience.