Deep Reinforcement Learning Queue Management for Undersea Networks
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Undersea acoustic networks face challenges in effective queue management due to limited bandwidth, high delays, and variability in the underwater environment, leading to network congestion and inefficient use of resources, which existing technologies fail to adequately address.
Innovation Solution
A computer-implemented method using deep reinforcement learning to dynamically manage network traffic by classifying packets based on quality-of-service requirements and allocating bandwidth through a weighted fair-queuing policy, which adapts to changing network conditions and prioritizes traffic types using a neural network and soft actor-critic algorithm.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If large memory buffers are deployed with each network node to mitigate the need for AQM policies, then network congestion is reduced, but computational requirements for managing individual queues increase and bufferbloat occurs which negatively impacts time-sensitive data traffic
Solution Approach 1:
The patent replaces traditional mechanical queue management mechanisms (buffer management, AQM policies) with a neural network-based intelligent system. The neural network dynamically determines optimal queue management decisions by learning from historical traffic patterns and network states, substituting complex computational queue management with an adaptive intelligent agent that reduces computational overhead while maintaining or improving congestion mitigation performance.
Solution Approach 2:
The patent dynamically changes queue management parameters (buffer allocation, packet dropping probabilities, scheduling weights) based on real-time network conditions and learned patterns. Instead of fixed buffer sizes and static AQM policies, the system adapts parameters continuously through neural network inference, allowing optimal performance across varying traffic conditions without requiring large fixed buffers or complex computational management.
2Ease of operation
If fair queueing (FQ) scheduling policies are used to manage network traffic, then bandwidth fairness is improved, but network-bandwidth resources are used suboptimally if the policies are not adapted to dynamically respond to changes in the level or resources available to the network
Solution Approach 1:
The patent transforms static fair queueing policies into dynamic adaptive policies through neural network integration. The system continuously learns from network traffic patterns, queue states, and bandwidth availability to dynamically adjust scheduling decisions. This allows the system to maintain bandwidth fairness while adapting to changing network conditions, preventing suboptimal resource utilization that occurs with rigid FQ policies.
Solution Approach 2:
The patent implements feedback mechanisms where the neural network continuously monitors network performance metrics (queue lengths, bandwidth utilization, delay statistics) and adjusts scheduling policies accordingly. This closed-loop control enables the system to respond to changing network conditions in real-time, maintaining fairness while optimizing overall bandwidth utilization based on actual network state and learned patterns.
Data Source
AI summary
A reinforcement-learning based queue-management method for undersea networks. In one embodiment, a computer-implemented method to shape traffic in a dynamic network, comprising measuring network states, classifying each of a plurality of packets into one of a plurality of traffic types based on quality-of-service requirements, defining a decision epoch interval, determining a policy with a plurality of neural networks and with a deep reinforcement learning agent that maximizes entropy-regularized long-term reward and produces a plurality of weighted fair-queuing weights based on an observation of the state of the queues at the decision epoch intervals, and using the policy to allocate predefined portions of the available transmission bandwidth across the plurality of traffic types by defining each type with a WFQ weight.