Sd-wan packet scheduler for QOE optimized capacity arbitrage

Abstract

Described herein are software-defined wide area network (SD-WAN) packet scheduling technologies across heterogeneous underlay networks having different latencies and capacities.

Description

VS2636-WO-1 PATENT SD-WAN PACKET SCHEDULER FOR QOE OPTIMIZED CAPACITY ARBITRAGECROSS-REFERENCE TO RELATED APPLICATION(S)

[0001] This application claims the benefit of priority to U.S. Provisional Application No. 63 / 765,238 filed February 28, 2025, which is expressly incorporated by reference herein in its entirety for all purposes.BACKGROUNDField

[0002] The present disclosure generally relates to scheduling network packets in networks, such as software-defined wide area networks.Description of Related Art

[0003] A software-defined wide area network (SD-WAN) is a technology that uses centralized software control to manage and optimize WAN connections across multiple network links (e.g., satellite, broadband, fiber, LTE / 5G, etc.). SD-WAN can use centralized control elements, such as a controller, to define policies and to steer network traffic intelligently. Whereas traditional WANs are hardwarecentric networks, SD-WANs are software-driven networks that are particularly advantageous for cloud and distributed applications.SUMMARY

[0004] In some aspects, the techniques described herein relate to a scheduler for a communications network that transmits data over a plurality of heterogeneous underlay networks, the scheduler including: a network interface for communicating with the plurality of heterogeneous underlay networks; a non-transitory computer-readable medium storing processor-executable instructions; and a processor communicatively coupled to the network interface and the non-transitory computer-readable medium, the processor-executable instructions configured to cause the processor to: rank the plurality of heterogeneous underlay networks from a highest priority to a lowest priority, the highest priority assigned to an underlay network of the plurality of heterogeneous underlay networks with a lower latency than the other underlay networks of the plurality of heterogeneousunderlay networks; analyze a packet for transmission over an underlay network of the at least one of the plurality of heterogeneous underlay networks to determine a flow to which the packet belongs; responsive to determining that the packet belongs to a new flow, send the packet to the underlay network assigned the highest priority; determining a behavior of the flow to which the packet belongs; responsive to the behavior corresponding to a continuous throughput less than a throughput threshold rate, send the packet to the underlay network assigned the highest priority; responsive to the behavior corresponding to a burst size less than a burst threshold amount, send the packet to the underlay network assigned the highest priority; and responsive to the behavior corresponding to a burst size greater than a burst threshold amount, send the packet to an underlay network assigned a lower priority lower than the highest priority.

[0005] In some aspects, the techniques described herein relate to a scheduler, wherein the plurality of heterogeneous underlay networks includes a low earth orbit (LEO) satellite network.

[0006] In some aspects, the techniques described herein relate to a scheduler, wherein the LEO satellite network corresponds to the underlay network with the lower latency than the other underlay networks of the plurality of heterogeneous underlay networks.

[0007] In some aspects, the techniques described herein relate to a scheduler, wherein the plurality of heterogeneous underlay networks includes a geosynchronous orbit (GEO) satellite network.

[0008] In some aspects, the techniques described herein relate to a scheduler, wherein the GEO satellite network has a higher latency than the LEO satellite network.

[0009] In some aspects, the techniques described herein relate to a scheduler, wherein the lowest priority is assigned to an underlay network of the plurality of heterogeneous underlay networks with a longer latency than the other underlay networks of the plurality of heterogeneous underlay networks.

[0010] In some aspects, the techniques described herein relate to a scheduler, wherein determining the packet belongs to a new flow includes determining that the packet belongs to an idle-timed-out connection flow.

[0011] In some aspects, the techniques described herein relate to a scheduler, wherein determining the behavior of the flow to which the packetbelongs includes employing a set of token bucket filters (TBFs) to assign a link priority to each packet of the flow, the link priority ranging from a highest link priority to a lowest link priority, the set of TBFs including a plurality of TBFs with a first TBF corresponding to the highest link priority and a second TBF corresponding to a link priority lower than the link priority of the first TBF.

[0012] In some aspects, the techniques described herein relate to a scheduler, wherein the behavior corresponds to the continuous throughput being less than the throughput threshold rate when the set of TBFs assigns the packet the link priority corresponding to the highest link priority.

[0013] In some aspects, the techniques described herein relate to a scheduler, wherein the behavior corresponds to the burst size being less than the burst threshold amount when the set of TBFs assigns the packet the link priority corresponding to the highest link priority.

[0014] In some aspects, the techniques described herein relate to a scheduler, wherein the behavior corresponds to the burst size being greater than the burst threshold amount when the set of TBFs assigns the packet the link priority corresponding to a lower link priority than the highest link priority.

[0015] In some aspects, the techniques described herein relate to a scheduler, wherein a number of token bucket filters in the set of TBFs is one less than a number of underlay networks of the plurality of heterogeneous underlay networks.

[0016] In some aspects, the techniques described herein relate to a scheduler, wherein a separate set of TBFs is used for each unique flow.

[0017] In some aspects, the techniques described herein relate to a scheduler, wherein a token fill rate of a first TBF of the set of TBFs corresponds to a stable throughput rate on the underlay network assigned the highest priority.

[0018] In some aspects, the techniques described herein relate to a scheduler, wherein an initial token size of the set of TBFs corresponds to a size sufficient to transmit initial burst data for the new flow to perform a slow start process.

[0019] In some aspects, the techniques described herein relate to a scheduler, wherein the set of TBFs is implemented by doing the following: set the first TBF as a current TBF to initiate a loop through the plurality of TBFs in the set of TBFs; responsive to determining that the current TBF has enough tokens to sendthe packet, assign to the packet the link priority corresponding to the current TBF; and responsive to determining that the current TBF does not have enough tokens to send the packet, set a next TBF as the current TBF to continue the loop, the next TBF corresponding to a TBF with a next highest link priority.

[0020] In some aspects, the techniques described herein relate to a scheduler, wherein the processor-executable instructions are further configured to cause the processor to responsive to determining that the underlay network assigned the highest priority is congested, send the packet to the underlay network assigned a next highest priority for packets determined to be sent to the underlay network assigned the highest priority.

[0021] In some aspects, the techniques described herein relate to a scheduler, wherein the processor-executable instructions are further configured to cause the processor to determine flow statistic information for the flow to which the packet belongs.

[0022] In some aspects, the techniques described herein relate to a scheduler, wherein the processor-executable instructions are further configured to cause the processor to implement a delay sensitivity classifier (DSC), the DSC configured to receive the packet and to set a delay-sensitivity level based on an analysis of the packet.

[0023] In some aspects, the techniques described herein relate to a scheduler, wherein the delay-sensitivity level corresponds to an underlay network of the plurality of heterogeneous underlay networks.

[0024] In some aspects, the techniques described herein relate to a scheduler, wherein the DSC is further configured to set the delay-sensitivity level using a committed bucket refilled using a committed information rate and a peak bucket refilled using a peak information rate.

[0025] In some aspects, the techniques described herein relate to a scheduler, wherein the DSC is further configured to set the delay-sensitivity level by determining whether a length of the packet exceeds a size of the committed bucket and the peak bucket.

[0026] In some aspects, the techniques described herein relate to a scheduler, wherein the DSC is further configured to set the delay-sensitivity level to a highest level responsive to determining that the length of the packet is not greater than the size of the committed bucket.

[0027] In some aspects, the techniques described herein relate to a scheduler, wherein the DSC is further configured to set the delay-sensitivity level to a lower level than the highest level responsive to determining that the length of the packet is not greater than the size of the peak bucket.

[0028] In some aspects, the techniques described herein relate to a scheduler, wherein the DSC is further configured to set the delay-sensitivity level to a lowest level responsive to determining that the length of the packet is greater than the size of the peak bucket.

[0029] In some aspects, the techniques described herein relate to a scheduler, wherein the packet is sent to an underlay network based at least in part on the delay-sensitivity level set by the DSC.

[0030] In some aspects, the techniques described herein relate to a scheduler for a communications network that transmits overlay traffic over a plurality of heterogeneous underlay bearer networks having different latencies, the scheduler including: a network interface configured to communicate with the plurality of heterogeneous underlay bearer networks; a non-transitory computer-readable medium storing processor-executable instructions; and a processor communicatively coupled to the network interface and the non-transitory computer-readable medium, the processor-executable instructions configured to cause the processor to: receive channel data frames corresponding to overlay packets associated with one or more flows; classify the channel data frames into delaysensitive frames and throughput-sensitive frames based at least in part on behavioral characteristics of the flows; execute a delay sensitivity round in which delay-sensitive frames are preferentially scheduled for transmission over at least one underlay bearer network having a lower latency than at least one other underlay bearer network of the plurality of heterogeneous underlay bearer networks; execute a throughput sensitivity round subsequent to the delay sensitivity round in which throughput-sensitive frames are scheduled for transmission over at least one underlay bearer network having a higher latency than the at least one lower-latency underlay bearer network; and execute a max bonded round subsequent to the throughput sensitivity round in which remaining throughput-sensitive frames are scheduled for transmission over any uncongested underlay bearer network of the plurality of heterogeneous underlay bearer networks to increase aggregate bonded capacity utilization, wherein schedulingdecisions in each round are performed based at least in part on congestion control state information associated with each underlay bearer network.

[0031] In some aspects, the techniques described herein relate to a scheduler, wherein the overlay traffic is transported between overlay endpoints using multi-path QUIC (MP-QUIC), and wherein the channel data frames correspond to QUIC stream frames encapsulated within bearer-specific QUIC connections established over the plurality of heterogeneous underlay bearer networks.

[0032] In some aspects, the techniques described herein relate to a scheduler, wherein each underlay bearer network corresponds to a distinct QUIC path of the MP-QUIC connection, and wherein congestion control state information includes a congestion window and pacing rate maintained independently for each QUIC path.

[0033] In some aspects, the techniques described herein relate to a scheduler, wherein during the delay sensitivity round, transport acknowledgment packets are preferentially scheduled over the at least one lower-latency underlay bearer network to reduce effective control loop delay of an overlay transport protocol.

[0034] In some aspects, the techniques described herein relate to a scheduler, wherein the transport acknowledgment packets include acknowledgment frames associated with a reliable overlay channel transported between overlay endpoints.

[0035] In some aspects, the techniques described herein relate to a scheduler, wherein the congestion control state information for each underlay bearer network includes at least one of: a transmission window size, a pacing rate, a measured round-trip time, a packet loss rate, or available transmission credits, and wherein scheduling in each round is conditioned on available transmission capacity indicated by the congestion control state information.

[0036] In some aspects, the techniques described herein relate to a scheduler, wherein the scheduler limits distribution of channel data frames belonging to a same overlay flow across underlay bearer networks having substantially different round-trip times to reduce inter-packet delay variation.

[0037] In some aspects, the techniques described herein relate to a scheduler, wherein the scheduler forwards channel data frames of a same overlayUDP flow to a same underlay bearer network except when required by congestion conditions identified during the max bonded round.

[0038] In some aspects, the techniques described herein relate to a scheduler, wherein within at least one of the delay sensitivity round, the throughput sensitivity round, or the max bonded round, channel data frames are selected for transmission according to a weighted fairness policy applied among multiple overlay flows.

[0039] In some aspects, the techniques described herein relate to a scheduler, wherein the weighted fairness policy is based at least in part on a channel priority value or a configured traffic weight associated with the overlay flow.

[0040] In some aspects, the techniques described herein relate to a scheduler, wherein the plurality of heterogeneous underlay bearer networks includes: a low earth orbit (LEO) satellite network having a first latency; and a geosynchronous earth orbit (GEO) satellite network having a second latency greater than the first latency and superior bandwidth economics than the LEO satellite network.

[0041] In some aspects, the techniques described herein relate to a scheduler, wherein the delay sensitivity round schedules delay-sensitive frames to the LEO satellite network and the throughput sensitivity round schedules throughput-sensitive frames to the GEO satellite network.

[0042] In some aspects, the techniques described herein relate to a scheduler, wherein acknowledgment packets associated with a bearer operating over the GEO satellite network are transmitted over the LEO satellite network to reduce an effective feedback delay associated with congestion control of the GEO satellite network.

[0043] In some aspects, the techniques described herein relate to an overlay network endpoint for a software-defined wide area network (SD-WAN) including: a plurality of bearer interfaces respectively associated with a plurality of heterogeneous access networks; a plurality of overlay channel modules configured to generate channel data frames corresponding to overlay UDP or TCP sessions; a congestion control module configured to maintain independent congestion control state for each bearer interface; and a round-based delay-sensitive classification and forwarding (DSC-F) scheduler configured to: in a first scheduling round, map delay-sensitive channel data frames to a bearer interface associatedwith a lower-latency access network; in a second scheduling round, map throughput-sensitive channel data frames to a bearer interface associated with a higher-latency access network with superior bandwidth economics; and in a third scheduling round, map remaining channel data frames to any bearer interface having available transmission capacity based on congestion control state, wherein the scheduler reduces inter-packet delay variation by limiting distribution of channel data frames from a same overlay flow across bearer interfaces having substantially different round-trip times.

[0044] In some aspects, the techniques described herein relate to an overlay network endpoint, wherein the overlay channel modules are configured to support multi-path QUIC (MP-QUIC) sessions between the overlay network endpoint and a corresponding remote endpoint, and wherein each bearer interface corresponds to a respective QUIC path of an MP-QUIC connection.

[0045] In some aspects, the techniques described herein relate to an overlay network endpoint, wherein the congestion control module maintains, for each QUIC path, a congestion window and a pacing rate that are independently updated based on path-specific feedback.

[0046] In some aspects, the techniques described herein relate to an overlay network endpoint, wherein the delay-sensitive channel data frames include transport acknowledgment frames, and wherein the scheduler preferentially maps the transport acknowledgment frames to the bearer interface associated with the lower-latency access network to reduce control loop delay.

[0047] In some aspects, the techniques described herein relate to an overlay network endpoint, wherein the transport acknowledgment frames correspond to acknowledgment messages of a reliable overlay channel transporting data frames between overlay network endpoints.

[0048] In some aspects, the techniques described herein relate to an overlay network endpoint, wherein the congestion control state for each bearer interface includes at least one of: a transmission window size, a pacing rate, a measured round-trip time, a packet loss rate, or available transmission credits, and wherein execution of the first, second, and third scheduling rounds is conditioned on available transmission capacity indicated by the congestion control state.

[0049] In some aspects, the techniques described herein relate to an overlay network endpoint, wherein the scheduler is configured to forward channeldata frames belonging to a same overlay UDP flow to a same bearer interface except when congestion state indicates insufficient transmission capacity.

[0050] In some aspects, the techniques described herein relate to an overlay network endpoint, wherein within at least one of the first scheduling round, the second scheduling round, or the third scheduling round, channel data frames are selected for transmission according to a weighted fairness policy applied among multiple overlay flows.

[0051] In some aspects, the techniques described herein relate to an overlay network endpoint, wherein the weighted fairness policy is based at least in part on a channel priority value or a configured traffic weight associated with each overlay flow.

[0052] In some aspects, the techniques described herein relate to an overlay network endpoint, wherein the scheduler is configured to identify a burst of packets associated with an overlay flow and to map a first predetermined number of bytes or packets of the burst to the bearer interface associated with the lower-latency access network during the first scheduling round.

[0053] In some aspects, the techniques described herein relate to an overlay network endpoint, wherein the plurality of heterogeneous access networks includes a low earth orbit (LEO) satellite network and a geosynchronous earth orbit (GEO) satellite network, the LEO satellite network having a lower round-trip time than the GEO satellite network.

[0054] In some aspects, the techniques described herein relate to an overlay network endpoint, wherein the scheduler maps delay-sensitive channel data frames to the bearer interface associated with the LEO satellite network and maps throughput-sensitive channel data frames to the bearer interface associated with the GEO satellite network.

[0055] In some aspects, the techniques described herein relate to an overlay network endpoint, wherein acknowledgment packets associated with a bearer operating over the GEO satellite network are transmitted over the LEO satellite network to reduce effective congestion feedback delay for the GEO satellite network.

[0056] In some aspects, the techniques described herein relate to a method of scheduling packets for transmission over a plurality of heterogeneous underlay networks in a communications network, the method including: ranking theplurality of heterogeneous underlay networks from a highest priority to a lowest priority, the highest priority assigned to an underlay network having a lower latency than other underlay networks of the plurality; analyzing a packet to determine a flow to which the packet belongs; responsive to determining that the packet belongs to a new flow, sending the packet to the underlay network assigned the highest priority; determining a behavior of the flow to which the packet belongs; responsive to the behavior corresponding to a continuous throughput less than a throughput threshold rate, sending the packet to the underlay network assigned the highest priority; responsive to the behavior corresponding to a burst size less than a burst threshold amount, sending the packet to the underlay network assigned the highest priority; and responsive to the behavior corresponding to a burst size greater than a burst threshold amount, sending the packet to an underlay network assigned a lower priority than the highest priority.

[0057] In some aspects, the techniques described herein relate to a method, wherein the plurality of heterogeneous underlay networks includes a low earth orbit (LEO) satellite network and a geosynchronous earth orbit (GEO) satellite network, the LEO satellite network having a lower latency than the GEO satellite network and the GEO satellite network having more economical capacity than the LEO satellite network. In some implementations, the GEO satellite network has a higher capacity than the LEO satellite network.

[0058] In some aspects, the techniques described herein relate to a method, wherein determining the behavior of the flow includes employing a set of token bucket filters (TBFs) to assign a link priority to the packet, the set of TBFs including a first TBF corresponding to a highest link priority and at least one additional TBF corresponding to a lower link priority.

[0059] In some aspects, the techniques described herein relate to a method, wherein a separate set of token bucket filters is maintained for each unique flow, and wherein a token fill rate of the first TBF corresponds to a stable throughput rate of the underlay network assigned the highest priority.

[0060] In some aspects, the techniques described herein relate to a method, further including, responsive to determining that the underlay network assigned the highest priority is congested, sending the packet to an underlay network assigned a next highest priority.

[0061] In some aspects, the techniques described herein relate to a method, further including implementing a delay sensitivity classifier (DSC) configured to assign a delay-sensitivity level to the packet using a committed bucket refilled at a committed information rate and a peak bucket refilled at a peak information rate, and sending the packet to one of the plurality of heterogeneous underlay networks based at least in part on the delay-sensitivity level.

[0062] In some aspects, the techniques described herein relate to a method of forwarding overlay traffic over a plurality of heterogeneous underlay bearer networks having different latencies, the method including: receiving channel data frames corresponding to overlay packets associated with one or more flows; classifying the channel data frames into delay-sensitive frames and throughputsensitive frames based at least in part on behavioral characteristics of the flows; executing a delay sensitivity scheduling round in which delay-sensitive frames are preferentially transmitted over at least one underlay bearer network having a lower latency than at least one other underlay bearer network of the plurality; executing a throughput sensitivity scheduling round subsequent to the delay sensitivity scheduling round in which throughput-sensitive frames are transmitted over at least one underlay bearer network having a higher latency than the at least one lower-latency underlay bearer network; and executing a max bonded scheduling round subsequent to the throughput sensitivity scheduling round in which remaining throughput-sensitive frames are transmitted over any uncongested underlay bearer network of the plurality to increase aggregate bonded capacity utilization, wherein scheduling in each round is performed based at least in part on congestion control state information associated with each underlay bearer network.

[0063] In some aspects, the techniques described herein relate to a method, wherein the overlay traffic is transported between overlay endpoints using multi-path QUIC (MP-QUIC), and wherein the channel data frames correspond to QUIC stream frames encapsulated within bearer-specific QUIC connections established over the plurality of heterogeneous underlay bearer networks.

[0064] In some aspects, the techniques described herein relate to a method, wherein each underlay bearer network corresponds to a distinct QUIC path of the MP-QUIC connection, and wherein the congestion control state information includes a congestion window and a pacing rate maintained independently for each QUIC path.

[0065] In some aspects, the techniques described herein relate to a method, wherein during the delay sensitivity scheduling round, transport acknowledgment packets are preferentially transmitted over the at least one lower-latency underlay bearer network to reduce effective control loop delay of an overlay transport protocol.

[0066] In some aspects, the techniques described herein relate to a method, wherein the transport acknowledgment packets include acknowledgment frames associated with a reliable overlay channel transported between overlay endpoints.

[0067] In some aspects, the techniques described herein relate to a method, wherein the congestion control state information for each underlay bearer network includes at least one of: a transmission window size, a pacing rate, a measured round-trip time, a packet loss rate, or available transmission credits, and wherein transmission in each scheduling round is conditioned on available transmission capacity indicated by the congestion control state information.

[0068] In some aspects, the techniques described herein relate to a method, further including limiting distribution of channel data frames belonging to a same overlay flow across underlay bearer networks having substantially different round-trip times to reduce inter-packet delay variation.

[0069] In some aspects, the techniques described herein relate to a method, further including selecting channel data frames for transmission within at least one of the delay sensitivity scheduling round, the throughput sensitivity scheduling round, or the max bonded scheduling round according to a weighted fairness policy applied among multiple overlay flows.

[0070] In some aspects, the techniques described herein relate to a method, wherein the plurality of heterogeneous underlay bearer networks includes a low earth orbit (LEO) satellite network having a first latency and a geosynchronous earth orbit (GEO) satellite network having a second latency greater than the first latency and superior bandwidth economics than the LEO satellite network.

[0071] In some aspects, the techniques described herein relate to a method, wherein acknowledgment packets associated with a bearer operating over the GEO satellite network are transmitted over the LEO satellite network to reduce effective congestion feedback delay associated with the GEO satellite network.

[0072] In some aspects, the techniques described herein relate to a method of operating an overlay network endpoint of a software-defined wide area network (SD-WAN), the overlay network endpoint being communicatively coupled to a plurality of heterogeneous access networks through a plurality of bearer interfaces, the method including: maintaining independent congestion control state for each bearer interface; generating channel data frames corresponding to overlay UDP or TCP sessions; in a first scheduling round, mapping delay-sensitive channel data frames to a bearer interface associated with a lower-latency access network; in a second scheduling round, mapping throughput-sensitive channel data frames to a bearer interface associated with a higher-latency access network with superior bandwidth economics; and in a third scheduling round, mapping remaining channel data frames to any bearer interface having available transmission capacity based on the congestion control state, wherein the method reduces inter-packet delay variation by limiting distribution of channel data frames from a same overlay flow across bearer interfaces having substantially different round-trip times.

[0073] In some aspects, the techniques described herein relate to a method, wherein the overlay UDP or TCP sessions are transported between the overlay network endpoint and a corresponding remote endpoint using multi-path QUIC (MP-QUIC), and wherein each bearer interface corresponds to a respective QUIC path of an MP-QUIC connection.

[0074] In some aspects, the techniques described herein relate to a method, wherein maintaining independent congestion control state for each bearer interface includes maintaining, for each QUIC path, a congestion window and a pacing rate that are independently updated based on path-specific feedback.

[0075] In some aspects, the techniques described herein relate to a method, wherein the delay-sensitive channel data frames include transport acknowledgment frames, and wherein mapping delay-sensitive channel data frames to the bearer interface associated with the lower-latency access network reduces control loop delay of an overlay transport protocol.

[0076] In some aspects, the techniques described herein relate to a method, wherein the transport acknowledgment frames correspond to acknowledgment messages of a reliable overlay channel transporting data frames between overlay network endpoints.

[0077] In some aspects, the techniques described herein relate to a method, wherein the congestion control state for each bearer interface includes at least one of: a transmission window size, a pacing rate, a measured round-trip time, a packet loss rate, or available transmission credits, and wherein execution of the first, second, and third scheduling rounds is conditioned on available transmission capacity indicated by the congestion control state.

[0078] In some aspects, the techniques described herein relate to a method, further including forwarding channel data frames belonging to a same overlay UDP flow to a same bearer interface except when congestion state indicates insufficient transmission capacity.

[0079] In some aspects, the techniques described herein relate to a method, further including selecting channel data frames for transmission within at least one of the first scheduling round, the second scheduling round, or the third scheduling round according to a weighted fairness policy applied among multiple overlay flows.

[0080] In some aspects, the techniques described herein relate to a method, wherein the plurality of heterogeneous access networks includes a low earth orbit (LEO) satellite network and a geosynchronous earth orbit (GEO) satellite network, the LEO satellite network having a lower round-trip time than the GEO satellite network.

[0081] In some aspects, the techniques described herein relate to a method, further including transmitting acknowledgment packets associated with a bearer operating over the GEO satellite network over the LEO satellite network to reduce effective congestion feedback delay associated with the GEO satellite network.

[0082] For purposes of summarizing the disclosure, certain aspects, advantages and novel features have been described herein. It is to be understood that not necessarily all such advantages may be achieved in accordance with any particular embodiment. Thus, the disclosed embodiments may be carried out in a manner that achieves or optimizes one advantage or group of advantages as taught herein without necessarily achieving other advantages as may be taught or suggested herein.BRIEF DESCRIPTION OF THE DRAWINGS

[0083] FIG. 1 illustrates an example communications network that includes a software-defined wide area network (SD-WAN) between a user terminal and a gateway that each include a per-flow throughput and burst-size aware packet forwarding (TBA-PF2) scheduler.

[0084] FIG. 2 illustrates another example communications network that includes a GEO satellite network and a LEO satellite network.

[0085] FIG. 3 illustrates another example communications network that implements delay sensitive classification and forwarding scheduling policies.

[0086] FIG. 4 illustrates a variety of network applications on a plot of bandwidth sensitivity vs. delay sensitivity.

[0087] FIG. 5 illustrates an example of network traffic with a TBA-PF2 scheduler on a communications network similar to the communications network of FIG. 2.

[0088] FIG. 6 illustrates an example of network traffic with a TBA-PF2 scheduler that employs a token bucket filter on the lowest latency link, the scheduler implemented on a communications network similar to the communications network of FIG. 2.

[0089] FIG. 7 illustrates an example of an implementation of a TBA-PF2 scheduler using token bucket filters.

[0090] FIG. 8 illustrates a flow and logic diagram of a behavior-based traffic classification system implemented in a TBA-PF2 scheduler.

[0091] FIGS. 9, 10, and 11 illustrate the throughput, as a function of time, for different underlay networks in a bulk download test.

[0092] FIG. 12 illustrates a graph of the throughput of the video traffic with the GEO sub-flow (black bars) and the LEO sub-flow (white bars) and a graph of the throughput on just the LEO link (white bars).

[0093] FIG. 13 illustrates an example delay sensitivity classifier (DSC) for use with TBA-PF2 packet forwarding technologies implemented in an SD-WAN scheduler in an SD-WAN that includes a plurality of underlay networks that include an LTE / LEO link, a MEO link, and a GEO link.

[0094] FIG. 14 illustrates an example implementation of a delay sensitive classifier for a TBA-PF2 packet scheduler or SD-WAN scheduler.

[0095] FIG. 15 illustrates a block diagram of a communication network implementing a SD-WAN design that includes a CPE proxy and an ISP proxy with SD-WAN packet forwarding technologies.

[0096] FIG. 16 illustrates an example of network traffic with a delay sensitive classification and forwarding (DSC-F) scheduler on a communications network similar to the communications network of FIG. 3.

[0097] FIG. 17 illustrates a block diagram of a communication network implementing a SD-WAN design that includes DSC-F packet forwarding technologies.

[0098] FIG. 18 illustrates a flow chart of an example method for performing delay sensitive classification and forwarding

[0099] FIG. 19 illustrates a block diagram of a scheduler of a communications system, such as a SD-WAN scheduler or TBA-PF2 packet forwarding module.DETAILED DESCRIPTION OF SOME EMBODIMENTS

[0100] The headings provided herein, if any, are for convenience only and do not necessarily affect the scope or meaning of the claimed subject matter.Overview

[0101] A software-defined wide area network (SD-WAN) is a virtual WAN architecture that allows Internet service providers (ISPs) to leverage a combination of access networks (e.g., cellular or long term evolution (LTE), broadband network service, satellite network service, etc.) to securely provide reliable connections for users and applications. Similarly, a software-defined local area network (SD-LAN) is a software-defined version of a traditional wired / wireless local area network, similar in concept to SD-WAN, applied inside a campus or enterprise network, for example. The SD-WAN itself operates as a virtual overlay network on top of multiple physical underlay networks. SD-WAN is useful for mobile Internet communications (or mobility SD-WAN services) where mobile terminals have access to two or more wireless networks such as geosynchronous orbit (GEO) satellite networks, medium Earth orbit (MEO) satellite networks, low Earth orbit (LEO) satellite networks, 4 / 5G wireless or cellular networks, and Wi-Fi networks. SD-WAN can stitch together more than one access network to provide networkcommunication to increase reliability and bandwidth. SD-WAN architectures are particularly advantageous where a single networks cannot cover an entire desired area so that multiple networks can be stitched together to provide targeted coverage of the desired area.

[0102] SD-WAN can utilize a variety of wireless networks or transport services to provide network communications and these wireless networks offer a wide range of characteristics such as capacity, transmission delay, bit error rate, operation cost, and the like. For example, GEO satellite networks offer more affordable connections compared to LEO satellite networks because LEO satellite networks require a greater number of satellites for the same service coverage area relative to GEO satellite networks. On the other hand, LEO satellite networks offer short round-trip time (RTT) (similar to RTT of 4 / 5G wireless networks (~50 ms)) while GEO satellite networks provide an order of magnitude higher RTT (~600 ms) relative to LEO satellite networks.

[0103] In some embodiments, an SD-WAN provides an overlay network abstraction over a plurality of heterogeneous underlay networks or links. These underlay networks can be provided by the wireless networks or transport services described herein. Overlay packets generated by end-user applications are encapsulated, tunneled, or otherwise transported over one or more available underlay networks. The overlay network thus abstracts the physical characteristics of the underlying access technologies, which may include terrestrial broadband, cellular networks, and satellite networks having differing latency, capacity, reliability, and cost characteristics.

[0104] Operation over multiple underlay networks requires a packet forwarding scheduler that determines, on a per-packet or per-flow basis, which underlay network is used to transport a given overlay packet. The scheduler therefore governs how overlay traffic is mapped to the available underlay resources in order to achieve performance and economic objectives. In general, SD-WAN scheduling paradigms can be categorized as switching policies, redundancy policies, and bonding policies, each of which provides different tradeoffs in terms of reliability, utilization efficiency, and quality of experience (QoE).

[0105] SD-WAN schedulers implement a variety of packet forwarding policies to achieve service goals over networks with heterogeneous quality-of-service (QoS) characteristics. Typical SD-WAN service goals over wirelessunderlay networks include improving packet delivery reliability, reducing cost, and / or improving user-perceived performance of the network (related to quality-of-experience or QoE). An SD-WAN scheduler, for example, may enact scheduling policies to decide how to schedule and direct packets. For example, an SD-WAN scheduler may enact policies that duplicate and forward each incoming packet to all the active underlay networks to maximize reliability. Another SD-WAN scheduler may give higher packet forwarding priority to underlay networks with lower operational costs to maximize yield. Yet another SD-WAN scheduler may choose the lowest delay path for each data packet to provide an optimal QoE to end user applications.

[0106] Switching policies typically forward overlay traffic to a primary underlay link while maintaining one or more secondary underlay links as standby or backup paths. When the primary link quality degrades beyond a threshold, traffic is switched to a backup link. Such approaches rely on monitoring underlay quality and generally do not utilize the aggregate capacity of multiple underlay networks simultaneously.

[0107] Redundancy policies duplicate overlay packets across multiple underlay links to create a delay-bounded fail-safe overlay path. The receiver maintains a resequencing buffer to suppress duplicate packets and reconstruct inorder delivery. Redundancy policies improve reliability and reduce delay variance but increase bandwidth consumption and are typically applied selectively to delaysensitive traffic rather than to all traffic.

[0108] Bonding policies forward overlay packets across multiple underlay networks concurrently in order to increase aggregate capacity utilization and meet performance objectives. Bonding policies may be further divided into flow-based forwarding and per-packet forwarding approaches. Flow-based bonding forwards all packets of a given 5-tuple flow to the same underlay network to avoid resequencing overhead. Per-packet bonding determines, for each packet, which underlay path to use based on scheduling criteria. While bonding offers improved utilization of heterogeneous links, it introduces additional challenges in hybrid environments where underlay networks have significantly different roundtrip times (RTTs) and capacity characteristics.

[0109] It is a typical goal for ISPs to provide a targeted QoE for their subscribers. To provide this targeted QoE in situations in which the user’sexperience is minimally or not impacted by latency or bandwidth constraints is a challenge for SD-WAN services, including mobility SD-WAN services, particularly so when these mobility SD-WAN services use GEO satellite or other high-latency networks along with low-latency satellite or wireless networks. Typical SD-WAN schedulers are designed and implemented for SD-WANs with homogenous links (e.g., links with similar capacity, latency, and operation cost).

[0110] Typical SD-WAN schedulers can be generally grouped into four categories. A first category of typical SD-WAN schedulers is redundancy-based schedulers that send duplicated traffic through all underlying links to achieve reliable packet delivery. However, these redundancy-based schedulers may reduce overall link utilization and yield a suboptimal QoE. Using such schedulers, SD-WAN link capacity is governed by the lowest capacity link resulting in suboptimal operation costs.

[0111] A second category of typical SD-WAN schedulers is fast-link selection schedulers that choose the underlay link having the lowest latency. This category of schedulers may provide a low-latency QoE with homogeneous underlay links (e.g., cellular networks from different operators). However, for hybrid satellite networks, especially for SD-WANs that include low-latency LEO satellite networks with high-capacity GEO satellite networks, the fast-link selection scheduler overloads the LEO links because of its low latency before starting to forward data to the GEO link with its higher latency. Thus, this scheduling policy does not efficiently use SD-WAN resources to achieve the targeted QoE.

[0112] A third category of typical SD-WAN schedulers is round robin load balancing or weighted round robin load balancing schedulers. These schedulers typically avoid link starvation but may result in poor performance for real-time applications (e.g., games). For example, for a hybrid satellite network with LEO and GEO links employing this type of load balancing scheduler, when data is sent using both LEO links and GEO links, the receiver must wait for the data to arrive from the slower link before sending it to the upper layer. In this case, from the perspective of the user, the SD-WAN is constrained by the higher latency of the GEO links and the benefit of the lower-latency LEO links is lost.

[0113] In hybrid SD-WAN deployments where underlay networks exhibit radically different RTTs, such as combinations of GEO satellite links, LEO satellite links, and terrestrial broadband or wireless links, naively distributing packetsacross these links may adversely impact end-to-end transport performance. Most overlay transport protocols, including transmission control protocol (TCP) and Quick user datagram protocol (UDP) Internet Connection (QUIC), employ congestion control algorithms that rely on RTT estimation for bottleneck bandwidth estimation, congestion window growth, and retransmission timeout calculations.

[0114] Blindly spreading packets from a single 5-tuple flow across both high-latency and low-latency underlay networks can result in bursty packet arrivals at the receiver, even when resequencing mechanisms ensure in-order delivery. Such burstiness may distort RTT measurements and impair congestion window growth, particularly during initial bandwidth probing phases such as slow start. As a result, overlay end-to-end transport performance may degrade despite the availability of aggregate underlay capacity.

[0115] A fourth category of typical SD-WAN schedulers is applicationbased schedulers that send packets to selected underlay links based on application type. To determine application type, the schedulers may use port-based classification or deep packet inspection (DPI) based application detection. However, port-based and DPI-based methods are not effective in modem networks because most Internet applications, including web game, video, and file sharing applications, use encrypted connections.

[0116] Modem Internet traffic is increasingly dominated by encrypted transport protocols, including Transport Layer Security (TLS) over TCP and QUIC over UDP. In particular, QUIC multiplexes multiple logical application streams within a single encrypted 5-tuple connection. As a consequence, flow-based bonding policies that operate at the granularity of a 5-tuple cannot distinguish between heterogeneous object types carried within a single encrypted connection, such as small delay-sensitive control objects and large bandwidth-intensive content objects. Because packet payloads are encrypted, deep packet inspection (DPI) techniques may not reliably classify traffic based on application-layer semantics. Thus, scalable SD-WAN scheduling solutions should operate effectively without reliance on payload inspection and should remain transportagnostic while accommodating multiplexed encrypted traffic patterns.

[0117] Accordingly, to overcome deficiencies of these categories of SD-WAN schedulers and to provide various advantages, disclosed herein are SD-WAN packet scheduling technologies and schedulers that implement behavioral-based traffic classification and / or delay sensitivity classification and forwarding. Behavioral-based traffic classification includes methods for classifying network traffic based on the behavior of the traffic which can include, for example and without limitation, rate, packet length, bursty behavior, and the like. Delay sensitivity classification includes methods for detecting delay sensitive applications, which can include real-time classification of applications, and forwarding includes methods for assigning higher priority to forward or transmit network traffic for applications identified as delay sensitive. The disclosed SD-WAN packet scheduling technologies are configured to be implemented in SD-WAN architectures that bond multiple network interfaces (such as wireless networks) with different characteristics as a single logical network interface to improve or maximize QoE to users, subscribers, or customers. Certain of these scheduling technologies implement per-flow throughput and burst-size aware packet forwarding (TBA-PF2) policies. Certain of these scheduling technologies implement delay sensitive classification and forwarding (DSC-F) policies. The disclosed SD-WAN schedulers provide a QoE close to that of a low-latency broadband network on a SD-WAN over various underlay networks with different latencies. TBA-PF2 schedulers use a behavioral-based traffic classification technique that monitors the size of a burst of each flow to assess a level of sensitivity to delay or throughput of the burst and forward packets according to the assessed sensitivity level to appropriate underlay networks. DSC-F schedulers apply technologies that identify delay sensitive packets including, for example and without limitation, initial packets belonging to a burst, low throughput streams, ack streams, and small HTTP objects. The DSC-F schedulers forward packets that are identified as delay sensitive over a low latency network and then forwards the rest of the traffic as throughput sensitive and forwards that traffic over higher latency and / or higher capacity networks with any overflow being sent over uncongested low latency networks.

[0118] In addition to achieving responsive user-perceived network speed, it is desirable to economically utilize substantially all available capacity across bonded access networks. Simply maximizing instantaneous throughput on a low-latency link may lead to inefficient use of higher-capacity, lower-cost underlay networks. Conversely, maximizing utilization of lower-cost links without regard to delay sensitivity may degrade QoE. Therefore, an effective SD-WAN schedulershould balance capacity utilization, latency sensitivity, and economic considerations while operating in a scalable manner across diverse traffic types, including encrypted transports. One or more of these goals is achieved with the disclosed SD-WAN technologies.

[0119] The SD-WAN schedulers disclosed herein successfully achieve one or more of the above goals based at least in part on the following observations. First, delay-sensitive network applications use small app data units. Based on the size of the app data units and their transmission intervals, a 5-tuple flow carrying the app data can be characterized as a continuous low throughput flow or multiple small bursts with some idle time in between the bursts. Examples of such applications that use small app data units include VoIP and First-Person Shooter (FPS) game applications because these apps frequently transmit small data units resulting in the carrying flow exhibiting relatively constant but small transmission rates. Other examples of such applications include Domain Name Service (DNS) that uses small datagrams to find IP addresses for a given hostname. This application is an example of a delay-sensitive application that exhibits small bursts followed by rather long idle times. World Wide Web (WWW) traffic can be categorized as a delay-sensitive or bandwidth-sensitive application based on the type of the objects used. A web application uses the initial HTTP transaction to download a small text-based HTML file and to parse it to build a page model. Then, the web application identifies additional objects including scripts, pictures and videos required to paint the page and initiates additional HTTP transactions to the web servers identified by the object URL. Most web browsers provide the first impression of the page to users before completing large picture or video object transfers. Therefore, HTTP transactions for HTML files, scripts and small icons are sensitive to network delay to provide a snappy web experience.

[0120] Second, a new 5-tuple flow is sensitive to network delay during its initial stages. This is because when a flow starts, transport layer (L4) signaling applications exchange small packets to negotiate the protocol (e.g., TCP, QUIC) parameters followed by TLS signaling applications validating certificates and negotiating the encryption keys before the connection is used by applications for the transmission of objects of various sizes.

[0121] Third, most 5-tuple flows are used or shared by multiple network applications for the lifetime of the flow. By way of example, HTTP transactions arethe most used network object transfer model on the Internet and include, for example and without limitation, web page and object transfers for web browsing and mobile apps or streaming video (which may account for more than about 70% of Internet traffic volume). After transport protocol and TLS parameter negotiation is completed, a HTTPS (e.g., HTTP over TLS) connection to a content provider (e.g., a front-end) server is reused for many different HTTP transactions to transfer various type of objects including HTML pages, scripts, pictures and videos. For example, a single L4 connection can be used for multiple HTTP transactions from the same server. Thus, most 5-tuple flows have various size bursts with different idle times in between the bursts.

[0122] Last, the delivery time for small objects (e.g., a HTTP transaction latency) is mainly influenced by RTT of the network while delivery time for large objects is mainly affected by the available bandwidth of the network path. When a new L4 connection is established, a slow start algorithm is used by the transport protocols to probe available capacity of the network path. The slow start algorithm doubles the transmission window size, referred to as the congestion window (CWND), every RTT. Typically, slow start algorithms start with a small initial congestion window size of 10 packets. Thereafter, the slow start algorithm may double the window size every RTT going from 10 to 20 to 40 to 80 packets, etc. Thus, using a low-latency underlay network can be a superior choice for forwarding traffic in a wireless SD-WAN environment to quickly increase the L4 transmission rate in the beginning of a flow.

[0123] Yet, as the L4 transmission window size increases, the impact of RTT on the transmission rate diminishes meaning that choosing a wider-band underlay network shortens the object delivery time even when the chosen network has a longer transmission delay. Thus, bursts are bandwidth sensitive rather than delay sensitive as the size increases.

[0124] In addition, transport protocols such as TCP and QUIC dynamically adjust their congestion window based on measured RTT and observed packet delivery characteristics. When packets are transmitted over multiple underlay networks with substantially different RTTs, RTT samples may vary widely, potentially leading to conservative congestion window growth or premature retransmission timeouts. In hybrid SD-WAN environments, careful packet scheduling is therefore required to prevent transport-layer performancedegradation arising from RTT heterogeneity while still leveraging the aggregate capacity of available underlay networks.

[0125] Accordingly, the disclosed SD-WAN packet forwarding policies are tailored to provide capacity arbitrage capabilities to economically use the available underlay networks in a geographical region resulting in superior or optimal QoE. Generally, the disclosed policies allow for packets from a flow to be forwarded to any available underlay network. In some implementations, scheduling policies apply the following to determine packet forwarding: 1) packets from new or idle-timed-out connections are forwarded to the lowest latency underlay network, 2) packets from a continuous low throughput flow up to a throughput threshold rate, Tbps, are forwarded to the lowest latency underlay network, 3) packets from a flow burst up to a burst threshold amount, S bytes, are forwarded to the lowest latency underlay network, and 4) packets from a flow burst after the burst threshold, S bytes, are forwarded to an underlay network with sufficient available capacity, where the underlay network is selected based at least in part on economics. For each of the policies selecting the lowest latency underlay network, if the lowest latency underlay network is in congestion, packets are forwarded to the next lowest latency underlay network(s), and so on. For policy 4, the network with the lowest delay can also be considered for the forwarding path based at least in part on the cost-per-bit economics.

[0126] In some implementations, the scheduling policies apply the following to determine packet forwarding based on delay sensitivity, where packets marked as delay sensitive are forwarded to the low latency network. The scheduling policy can be implemented as follows: 1) identify a 5-tuple burst and mark the first N-bytes or the first N-packets as delay sensitive, 2) identify a continuous low-throughput 5-tuple flow stream and mark the packets of the stream as delay sensitive, 3) identify overlay transport ack streams and mark the packets as delay sensitive to improve or optimize overlay transport protocol control loop delay, 4) identify small HTTP objects and mark the associated packets as delay sensitive. All other packets or network traffic is marked as throughput sensitive (or not delay sensitive) and forwarded to higher latency networks (e.g., networks with higher latency than the low latency network). If the high latency underlay link(s) is congested, overflow traffic is directed back to uncongested low latency links to more fully utilize the bonded underlay network capacity.

[0127] The disclosed technologies are configured to leverage a relatively small amount of low-latency capacity to provide a customer experience similar to a network with a large amount of low-latency capacity. This is done by employing a behavioral-based traffic classification technique that monitors throughput and burst size for each flow to assess the level of delay sensitivity and / or bandwidth sensitivity of the bursts and forward packets to appropriate underlay networks according to the level of sensitivity. The disclosed technologies are particularly useful in providing a user with a high-quality Internet experience where the user is, for example, browsing the web, streaming video, playing online games, etc. on a network that uses heterogeneous underlay transport services. Heterogeneous underlay transport services are transport services with different network characteristics such as, for example and without limitation, latency, bandwidth, bit error rate, etc. The disclosed technologies use capacity arbitrage to economically use heterogeneous underlay networks to provide an experience similar to that of a low-latency broadband network where the underlay networks only include only a relatively small amount of capacity on low-latency networks. By way of example, the heterogeneous underlay networks can include a GEO satellite network with a MEO satellite network, a LEO satellite network, 4 / 5G cellular networks, and / or WiFi networks. The underlay networks can have different characteristics and service prices.

[0128] The SD-WAN architectures described herein provide network communications for users by stitching together a plurality of heterogeneous transport services. From the perspective of the Internet service provider (ISP), the speed of the network, including the SD-WAN portion of the network, can be quantified using available bandwidth (e.g., bps measured using speed tests) or network path delay (e.g., RTT measured using pings). However, from the user’s point of view, the user quantifies perceived network speed in terms of the amount of time required to accomplish tasks that matter to the user. Thus, the disclosed TBA-PF2 technologies are configured to improve the user’s perception of network speed by intelligently leveraging underlay networks with different characteristics (e.g., capacity, bandwidth, latency, cost, etc.) to reduce the amount of time required to accomplish typical user tasks.

[0129] Typical SD-WAN schedulers use deep packet inspection-based (DPI) classification techniques to identify 5-tuple (i.e., source IP, source port,destination IP, destination port and protocol ID) application flows, divide the application flows into delay-sensitive and bandwidth-sensitive flows, and forward their packets to low- and high-latency networks, respectively. However, DPI-based classification and forwarding methods have drawbacks. For example, DPI-based methods typically do not work for encrypted traffic such as HTTPS, which is a dominant form of network traffic on the Internet. As another example, DPI-based methods rarely or infrequently perform flow re-classification due to the large computing overhead involved. This can lead to flow misclassification because a 5-tuple flow can be shared by multiple network applications.

[0130] In some implementations, the disclosed SD-WAN schedulers implement a DSC-F framework that operates at the packet level to determine, independent of payload inspection, whether a packet exhibits delay sensitivity or bandwidth sensitivity. The DSC-F framework assigns a delay-sensitivity level to packets based on behavioral characteristics observable at the network and transport layers, such as burst dynamics, throughput patterns, and transport state transitions. Packets classified as delay-sensitive are preferentially forwarded to lower-latency underlay networks, while packets classified as bandwidth-sensitive are forwarded to underlay networks selected based on available capacity and economic considerations. By operating at packet granularity rather than strictly at the 5-tuple flow level, DSC-F enables efficient bonding across heterogeneous underlay networks while mitigating transport-layer performance degradation arising from RTT asymmetry.

[0131] In certain embodiments, the SD-WAN overlay utilizes multi-path QUIC (MP-QUIC) as a transport substrate for tunneling and multi-path coordination. Because QUIC operates in user space, provides encrypted multiplexed streams over a single connection, and exposes transport state information that can be leveraged for path management and congestion awareness, MP-QUIC facilitates tighter integration between transport behavior and SD-WAN packet scheduling decisions. Compared to multi-path TCP (MPTCP), MP-QUIC enables flexible path establishment, stream-level multiplexing without head-of-line blocking, and transport-layer agility that improves responsiveness in hybrid RTT environments. The use of MP-QUIC in conjunction with DSC-F provides improved scalability, compatibility with encrypted traffic, and enhanced control over packet distribution across heterogeneous underlay networks.

[0132] In addition, described herein are technologies directed to SD-WAN packet scheduling that implement awareness of per-flow throughput and burst size to intelligently forward packets to appropriate transport services. The disclosed technologies are configured to 1) classify network packets by determining, at a packet level, which packets are delay sensitive, and 2) forward packets to appropriate underlay networks based on the classification of the packet. That is, the disclosed SD-WAN packet schedulers classify delay-sensitive packets and send at least some of such packets on low-latency underlay networks to reduce or minimize the impact of RTT on object delivery time. The disclosed SD-WAN packet scheduling technologies, which can include DSC-F policies or TBA-PF2 policies, may be particularly useful in mobility applications.

[0133] The disclosed scheduling technologies implement processes and systems that classify network traffic according to, among other things, a level of delay or bandwidth sensitivity. As described herein, delay-sensitive flows can be associated with delay-sensitive network applications, which can be characterized by transmission levels (e.g., small bursts or continuous low-level throughput), or with new flows while protocol and connection parameters are being negotiated. Advantageously, the disclosed scheduling technologies provide a light-weight, behavioral-based packet classification design that eliminates the need for deep packet inspection or a complex flow burst detection algorithm.

[0134] Advantageously, the disclosed scheduling technologies do not use DPI-based methods for scheduling packets, can function with encrypted traffic, and can perform flow re-classification. Typical SD-WAN schedulers have difficulty supporting delay sensitive applications. For example, an issue for typical SD-WAN schedulers operating in networks with heterogeneous RTT is that the scheduler can send delay sensitive traffic over links with long RTT, degrading QoE as a result. In contrast, the disclosed SD-WAN schedulers are designed to efficiently forward delay sensitive traffic over low latency networks while directing throughput sensitive traffic over longer latency and / or networks with more economical capacity (which may include networks with higher capacity).

[0135] The disclosed schedulers and scheduling technologies enhance QoE for all network traffic in a hybrid SD-WAN (e.g., an SD-WAN with heterogeneous networks). In certain implementations, the scheduling technologies advantageously reduce the end-to-end RTT by about half by sending L4acknowledgements to low-latency or low-delay networks. In some implementations, the disclosed scheduling technologies advantageously provide performance across the network similar to a performance enhancing proxy (PEP) using QUIC or VPN technologies by quickly growing the slow-start congestion window (CWND) using a low-latency network. In various implementations, the disclosed scheduling technologies advantageously enhance delivery of small objects using per-flow burst-based path selection.

[0136] Advantageously, the disclosed scheduling technologies provide capacity arbitrage in which the schedulers can balance QoE levels and operation costs in a situation in which there is limited low-latency network capacity. This can be accomplished, for example, by calibrating a burst size limit, S bytes, and a throughput rate limit, Tbps, for the lowest latency forwarding parameters. This allows the networks to economically and intelligently utilize available capacity by forwarding flow bursts that are in a bandwidth-sensitive state to the most economical uncongested network in the region.

[0137] In certain embodiments, the SD-WAN overlay is implemented using multi-path QUIC (MP-QUIC) as a transport substrate between SD-WAN edge endpoints (SEEP) and SD-WAN datacenter endpoints (SDEP). In these embodiments, QUIC over UDP is used as a tunneling protocol to establish multiple coordinated bearer paths across heterogeneous underlay networks, enabling encrypted, multiplexed transport of overlay traffic without reliance on deep packet inspection. Because QUIC operates in user space and supports stream multiplexing without head-of-line blocking, MP-QUIC enables flexible path establishment, rapid connection setup, and fine-grained congestion control per bearer. When integrated with delay sensitivity classification and forwarding (DSC-F), MP-QUIC allows delay-sensitive packets — such as control acknowledgments, small bursts, and low-throughput streams — to be preferentially forwarded over lower-latency underlay links, while throughput-sensitive traffic is directed to longer latency links with superior bandwidth economics, and possibly higher capacity, thereby improving end-to-end transport performance, reducing control loop delay in hybrid high- and low-latency environments, and maximizing utilization of bonded underlay capacity. Although multi-path TCP (MPTCP) may be used in various implementations, the use of MP-QUIC provides improved scalability, compatibilitywith encrypted traffic, and enhanced integration between transport-layer behavior and SD-WAN scheduling decisions.Example SD-WAN Architectures with Delay Sensitive Packet Schedulers

[0138] FIG. 1 illustrates an example communications network 100 that includes a SD-WAN 102 between a user terminal 110 and a gateway 150 that each include a SD-WAN scheduler. The SD-WAN 102 bonds multiple links 140 to provide communication between the user terminal 110 and the gateway 150. The links 140 can be heterogeneous transport services or networks, such as wireless networks. The user terminal 110 includes a client-side SD-WAN scheduler 115 and the gateway 150 includes a server-side SD-WAN scheduler 155. The SD-WAN schedulers 115, 155 are each configured to classify network traffic to be transmitted over the SD-WAN 102 based on delay sensitivity and / or bandwidth sensitivity and then forward packets to appropriate links of the links 140 based on the classification. In some implementations, the user terminal 110 does not include the client-side SD-WAN scheduler 115. In some implementations, the gateway 150 does not include the server-side SD-WAN scheduler 155.

[0139] As described herein, classification of network traffic by the SD-WAN schedulers 115, 155 can be done using awareness of per-flow throughput and burst size and / or using delay sensitivity classification heuristics. These classification techniques can be considered behavioral-based network classification or delay sensitivity classification and are different from typical scheduling technologies, as described herein. For example, in some implementations, the SD-WAN schedulers 115, 155 do not employ redundancybased scheduling, do not use fast-link selection scheduling, do not use round robin load balancing, or do not use application-based scheduling (e.g., using DPI-based classification or port-based classification), as described herein.

[0140] Rather, the SD-WAN schedulers 115, 155 use a behavioralbased traffic classification technique that monitors throughput and burst sizes of each flow to assess a level of sensitivity to delay or throughput of the burst and forwards packets according to the assessed sensitivity level to appropriate underlay networks of the links 140. For example, the SD-WAN schedulers 115, 155 forward to the lowest latency underlay network 1) packets from new (or idle-timed-out) connections, 2) packets from a continuous low throughput (e.g., flowsless than T bps), 3) packets from a flow burst less than S bytes, 4) packets from overlay ack streams, or 5) packets from small HTTP objects. For flow bursts greater than S bytes, the first N-bytes or packets are forwarded to the lowest latency network and the remaining traffic is forwarded to an underlay network with sufficient capacity. By implementing these policies, the SD-WAN schedulers 115, 155 provide a QoE close to that of a low-latency broadband network on a SD-WAN over various underlay networks with different latencies.

[0141] As set forth herein, TBA-PF2 policies provide two basic parameters for deciding which packets to forward to the lowest-latency network: burst size limit, S, and throughput limit, T. These parameters can be configured to balance QoE and the number of customers to support given limited capacity on low-latency networks. For a flow burst in a bandwidth sensitive state, TBA-PF2 policies provide the freedom to economically choose any available networks in the region for the user based on their cost per bit.

[0142] As set forth herein, DSC-F policies implement a heuristic approach to identifying delay sensitive traffic. The DSC-F policies are configured to manage bursts by identifying a 5-tuple burst and marking the first N bytes or the first N packets as delay sensitive to be forwarded to the low latency network. The DSC-F policies are configured to manage low throughput streams by identifying a continuous low-throughput 5-tuple flow stream and marking the packets of the stream as delay sensitive to be forwarded to the low latency network. This satisfies the goal of providing low latency for low-bitrate applications. The DSC-F policies are configured to improve or optimize overlay transport protocol control loop delays by identifying overlay transport ack streams and marking the packets as delay sensitive to be forwarded to a low latency network. The DSC-F policies are configured to manage small HTTP objects by identifying such objects and marking the associated packets as delay sensitive to be forwarded over the low latency network. This satisfies the goal of providing user-perceived web responsiveness. Packets that are not marked as delay sensitive are then marked as throughput sensitive or bandwidth sensitive and forwarded to a higher latency network until that network is congested. Once congested, overflow traffic is directed back to uncongested low latency links.

[0143] FIG. 2 illustrates another example communications network 200 that includes a GEO satellite network 240a and a LEO satellite network 240b. TheGEO satellite network 240a and the LEO satellite network 240b can be used as underlay networks in an SD-WAN overlay to provide network communication between a network 260 (e.g., the Internet) or a server 270 and customer premises equipment 211 (or CPE). The SD-WAN can be defined between a terminal 210 connected to the CPE 211 and a core network (CN) node 250 (e.g., a gateway) connected to the network 260. The communications network 200 can be referred to as a hybrid SD-WAN because the underlay networks are heterogeneous. The terminal 210 is similar to the user terminal 110 of FIG. 1. Similarly, the CN nodes 250 are similar to the gateway 150 of FIG. 1. Likewise, the underlay networks 240a, 240b are similar to the links 140 of FIG. 1.

[0144] The communications network 200 is configured to manage network traffic between the CPE 211 and the network 260 or server 270. The communications network 200 includes the terminal 210 that provides communication between the CPE 211 and the underlay networks 240a, 240b. Examples of terminals include user terminals and mobile terminals. In some embodiments, the communications network 200 is a combination of terrestrial, wireless, and / or satellite communications systems. The communications network 200 includes the CN nodes 250 that provide communication between the network 260 and the underlay networks 240a, 240b.

[0145] The disclosed TBA-PF2 technologies can be implemented in a scheduler. Such schedulers can be in communication with or part of any suitable network component such as a TBA-PF2 scheduler 215 that communicates with or is part of the terminal 210 and a TBA-PF2 scheduler 255 that communicates with or is part of the CN nodes 250.

[0146] The TBA-PF2 schedulers 215, 255 are configured to manage network resources of the underlay networks 240a, 240b. To send data to the external network 260 from the CPE 211, the TBA-PF2 scheduler 215 allocates network resources to the CPE 211, creating a transmission schedule of network traffic. Then, based on the schedule, the CPE 211 transmits data using the allocated resources. Similarly, to send data from the network 260 to the CPE 211, the TA-PF2 scheduler 255 allocates network resources to network devices and components configured to transmit data over the SD-WAN (comprising the underlay networks 240a, 240b) to the CPE 211, creating a schedule of transmission for those devices. The TBA-PF2 scheduler 215 can be configured toallocate network resources to flows from the CPE 211 and / or the TBA-PF2 scheduler 255 can be configured to allocate network resources to flows destined for the CPE 211.

[0147] The CN nodes 250 are configured to direct network traffic between the CPE 211 and the network 260. The CN nodes 250 can receive network traffic from network applications and direct the received network traffic to a targeted destination in the network 260 (e.g., the server 270). Similarly, the CN nodes 250 can receive network traffic from the network 260 and direct the network traffic to the CPE 211.

[0148] The communications network 200 may utilize various network architectures that include space and ground segments. For example, the space segment may include one or more satellites, while the ground segment may include one or more satellite user terminals, gateway terminals, network operations centers (NOCs), satellite and gateway terminal command centers, and / or the like. Some of these elements are not shown in the figure for clarity. The GEO satellite network 240a includes a geosynchronous earth orbit (GEO) satellite 205 or satellites and the LEO satellite network 240b includes a low earth orbit (LEO) satellite 206 or satellites. It is to be understood that either satellite network can be replaced with a medium earth orbit (MEO) satellite or satellites, It is to be understood that additional wireless networks can be added to the communications system to provide additional underlay networks, such as cellular networks, Wi-Fi networks, and / or a medium earth orbit (MEO) satellite or satellites.

[0149] The CPE 211 can include a router and can be configured to receive data to be routed over the communications network 200. The CPE 211 can include any type of consumer premises or mobile equipment (e.g., a telephone, modem, router, computer, set-top box, and the like). The terminal 210 is configured to route data to the underlay networks 240a, 240b (via respective customer satellite transceivers 220a, 220b). The underlay networks 240a, 240b include a forward link for sending information from the CN nodes 250 to the CPE 211 , and a return link for sending information from the CPE 211 to the CN nodes 250. The forward link includes a transmission path from the CN nodes 250 through a satellite transceivers 230a, 230b, through a respective satellite 205, 206 via satellite uplink channels, to the respective customer satellite transceivers 220a, 220b via satellite downlink channels, and to the CPE 211. The return link includes a transmissionpath from the customer satellite transceivers 220a, 220b, to the respective satellites 205, 206 via the satellite uplink channels, to the respective satellite transceivers 230a, 230b via the satellite downlink channels, and to the CN nodes 250. Each transmission channel may utilize multiple satellites and transceivers. In some implementations, part or all of the CN nodes 250 and / or the TBA-PF2 schedulers 215, 255 can be located in a virtual device residing in a public or private computing cloud.

[0150] The disclosed scheduling technologies function in any suitable communications system. For example, the communications system can be provided by satellites, by terrestrial-based equipment, or a combination of satellites and terrestrial networks. Thus, the concepts disclosed herein regarding TBA-PF2 packet forwarding techniques can be applied to network traffic and flows provided by any variety of communications systems.

[0151] FIG. 2 also illustrates a block diagram of communication elements between TCP endpoints. For example, the CPE 211 acts as a TCP endpoint 201 for communication with the server 270, which also acts as a TCP endpoint 204. Between the TCP endpoints 201 , 204, there is a customer-side proxy CPE proxy 202 (e.g., provided by the terminal 210) and an ISP proxy 203 (e.g., provided by the ISP gateway or CN nodes 250). There is TCP communication between the TCP endpoint 201 and the CPE proxy 202 as well as between the TCP endpoint 204 and the ISP proxy 203, respectively referred to as TCP #1 and TCP #3. The CPE proxy 202 and the ISP proxy act as common endpoints for multiple links, such as the underlay networks 240a, 240b. The CPE proxy 202 and the ISP proxy 203 implement multi-path TCP (MP-TCP), the disclosed packet forwarding technologies can be implemented using MP-QUIC or other tunneling protocols. TBA-PF2 packet forwarding technologies can be implemented in any SD-WAN tunneling protocols including MP-TCP proxies for TCP traffic, and MP-VPN or MP-QUIC proxies for end-to-end TCP and UDP traffic.

[0152] The TBA-PF2 schedulers 215, 255 manage sub-flows over each underlay network 240a, 240b. The LEO satellite network 240b has a much lower latency than the GEO satellite network 240a so that the TBA-PF2 schedulers 215, 255 assign the highest priority link (e.g., Lo as referred to herein) to the LEO link provided by the LEO satellite network 240b. In some implementations, the TBA-PF2 schedulers 215, 255 rank the plurality of heterogeneous underlay networksfrom a highest priority to a lowest priority, the highest priority assigned to an underlay network of the plurality of heterogeneous underlay networks with a lower latency than the other underlay networks of the plurality of heterogeneous underlay networks.

[0153] FIG. 3 illustrates another example communications network 300 that is similar to the communications network 200 of FIG. 2 where components with the same reference number refer to the same components (e.g., the GEO satellite network 240a and the LEO satellite network 240b). The communications network 300 implements DSC-F scheduling policies and utilizes MP-QUIC technologies.

[0154] As in the communications network 200, the GEO satellite network 240a and the LEO satellite network 240b can be used as underlay networks in an SD-WAN overlay to provide network communication between the network 260 (e.g., the Internet) or the server 270 and CPE 211. The SD-WAN can be defined between the terminal 210 connected to the CPE 211 and the CN node 250 (e.g., a gateway) connected to the network 260. The communications network 300 can be referred to as a hybrid SD-WAN because the underlay networks are heterogeneous.

[0155] The communications network 300 is configured to manage network traffic between the CPE 211 and the network 260 or the server 270. The communications network 300 includes a SEEP device 312 (where SEEP stands for an SD-WAN edge endpoint) and an SDEP device 352 (where SDEP stands for SD-WAN datacenter endpoint). The SEEP device 312 includes a client-side DSC-F scheduler 315 for managing traffic over the underlay networks between the CPE 211 and the server 270. The SDEP device 352 includes a server-side DSC-F scheduler 355 for managing traffic over the underlay networks between the SDEP device 352 and the SEEP device 312. As described in greater detail herein, the SEEP device 312 and the SDEP device 352 include components or modules for establishing TCP connections (e.g., using MPTCP) and / or for transmitting UDP packets (e.g., using MP-QUIC).

[0156] The disclosed DSC-F technologies can be implemented in a scheduler. Such schedulers can be in communication with or part of any suitable network component such as the client-side DSC-F scheduler 315 that communicates with or is part of the SEEP device 312 and the server-side DSC-F scheduler 355 that communicates with or is part of the SDEP device 352.

[0157] The DSC-F schedulers 315, 355 are configured to manage network resources of the underlay networks 240a, 240b. To send data to the external network 260 from the CPE 211, the client-side DSC-F scheduler 315 allocates network resources to the CPE 211, creating a transmission schedule of network traffic. Then, based on the schedule, the CPE 211 transmits data using the allocated resources. Similarly, to send data from the network 260 to the CPE 211 , the server-side DSC-F scheduler 355 allocates network resources to network devices and components configured to transmit data over the SD-WAN (comprising the underlay networks 240a, 240b) to the CPE 211, creating a schedule of transmission for those devices. The client-side DSC-F scheduler 315 can be configured to allocate network resources to flows from the CPE 211 and / or the server-side DSC-F scheduler 355 can be configured to allocate network resources to flows destined for the CPE 211.

[0158] The terminal 210 is configured to forward data to the underlay networks 240a, 240b (via respective customer satellite transceivers 220a, 220b) as determined by the client-side DSC-F scheduler 315. The underlay networks 240a, 240b include a forward link for sending information from the CN nodes 250 to the CPE 211 , and a return link for sending information from the CPE 211 to the CN nodes 250. The forward link includes a transmission path via satellite uplink channels and satellite downlink channels and a return link that includes a transmission path via the satellite uplink channels and satellite downlink channels. Each transmission channel may utilize multiple satellites and transceivers. In some implementations, part or all of the DSC-F schedulers 315, 355 can be located in a virtual device residing in a public or private computing cloud.

[0159] The disclosed scheduling technologies function in any suitable communications system. For example, the communications system can be provided by satellites, by terrestrial-based equipment, or a combination of satellites and terrestrial networks. Thus, the concepts disclosed herein regarding DSC-F packet forwarding techniques can be applied to network traffic and flows provided by any variety of communications systems.

[0160] FIG. 3 also illustrates communication paths between the CPE 211 and the destination server 270. Between the CPE 211 and the SEEP device 312, there can be a plurality of UDP and TCP sessions. Between the SEEP device 312 and the SDEP device 352, a plurality of bearers can provide L4 connections. Eachbearer can be an L4 connection between the SEEP device 312 and the SDEP device 352, which may be encrypted. In some implementations, there is one bearer per underlay network, but more than one bearer can be a plurality of bearers per underlay network (e.g., one bearer per service flow). Data frames from communication channels or streams can be inserted into bearer frames and transmitted to the endpoint on the other side of the access networks. Between the SDEP device 352 and the destination server 270, there can be a plurality of UDP and TCP sessions. For TCP connections, the SEEP device 312 and the SDEP device 352 can include TCP endpoints and / or proxies. In some implementations, MP-QUIC can be used as an end-to-end tunneling protocol between the CPE 211 and the server 270, as described in greater detail herein.

[0161] The DSC-F schedulers 315, 355 manage sub-flows over underlay networks 240a, 240b. The LEO satellite network 240b has a much lower latency than the GEO satellite network 240b so that the DSC-F schedulers 315, 355 assign delay sensitive packets to the LEO satellite network 240b and assign throughput sensitive packets to the GEO satellite network 240b until the GEO satellite network is congested where the DSC-F schedulers 315, 355 can forward overflow packets to the LEO satellite network 240b.

[0162] In some embodiments, overlay UDP flows transported using encryption (e.g., QUIC over UDP) do not permit Layer 4 (L4) termination within the SD-WAN overlay. Because payload inspection and transport termination are not available for such encrypted UDP flows, the SEEP and SDEP utilize a per-packet bonding scheduler operating in conjunction with the delay sensitivity classification and forwarding (DSC-F) policy to control how packets are mapped to underlay bearers. The scheduler reduces unnecessary forwarding path changes for packets belonging to the same overlay flow in order to minimize end-to-end inter-packet delay variation (jitter). In particular, DSC-F identifies burst behavior of a 5-tuple flow and forwards the first N-bytes or first N-packets of a burst to a low-latency underlay network, while forwarding subsequent packets of the burst to a higher-latency underlay network with superior bandwidth economics, and potentially higher capacity, until that network becomes congested. If the higher-latency underlay becomes congested, overflow packets are dynamically redirected to uncongested low-latency underlay networks so as to fully utilize bonded underlay capacity while maintaining delay performance objectives.

[0163] For end-to-end TCP sessions, SEEP and SDEP may perform TCP termination at the SD-WAN entrance points using a pair of TCP proxies or TCP termination points. Each end-to-end TCP 5-tuple flow is terminated at the SEEP and SDEP and mapped to a reliable SD-WAN channel. The reliable channel transports TCP payload across one or more bearers between SEEP and SDEP without data loss, thereby decoupling the end-to-end TCP congestion control loop from underlay path heterogeneity. This architecture allows the SD-WAN to optimize bearer selection and congestion control independently of the original end-to-end TCP session.

[0164] In various embodiments, each bearer established between SEEP and SDEP includes an independent congestion control loop that dynamically adapts to available capacity fluctuations of the corresponding underlay network. For example, for a bearer operating over a GEO or other high-latency network, the SD-WAN bearer congestion control can track available bandwidth while avoiding excessive packet loss or buffer bloat. Channel data frames generated by reliable or unreliable channels are inserted into bearer frames and transmitted according to the bearer congestion control rate. In some implementations, bearer congestion control loop delays are reduced by transmitting acknowledgment packets over lower-latency networks, thereby shortening feedback delay in hybrid high- and low-latency environments.

[0165] To reduce overlay packet delivery jitter, SEEP and SDEP avoid spreading data frames from a single end-to-end UDP channel across multiple bearers having substantially different RTTs, except when necessary for overflow or congestion management. For new QUIC 5-tuple flows, SEEP and SDEP forward the first N-bytes or N-packets over an available low-latency network to accelerate growth of the end-to-end congestion window, thereby enabling the overlay transport layer to more rapidly utilize available underlay capacity. Similarly, for newly detected 5-tuple bursts, the first N-bytes or N-packets may be forwarded over a low-latency network to reduce transmission delay of small HTTP requests and responses. In some embodiments, heuristic approaches may be used to detect new 5-tuple bursts based on inter-arrival time, packet count, or throughput characteristics.

[0166] SEEP and SDEP may further identify continuous low-throughput delay-sensitive flows, such as Voice-over-IP (VoIP) streams or first-person-shooter(FPS) game control flows, and preferentially forward such traffic over low-latency networks. Identification of delay-sensitive 5-tuple flows may be performed using heuristic traffic characterization techniques and / or shallow packet inspection methods that do not require decryption of encrypted payloads. When total traffic demand exceeds the capacity of higher-latency bearers, SEEP and SDEP may utilize all available bearer capacity by forwarding overflow packets to uncongested or uncontended low-latency networks, even when those packets are not otherwise designated as delay-sensitive, thereby maximizing bonded underlay capacity utilization.

[0167] In some embodiments, SEEP and SDEP further support passive characterization of bearer performance and end-to-end overlay performance. For example, each bearer may periodically measure and report usage, available capacity, packet loss rate, and speed availability during bearer reporting epochs (which may be greater than or equal to approximately five seconds, for example). Additionally, SEEP and SDEP may measure transmission completion time of channel bursts and generate statistical summaries, such as completion time per burst size for TCP or UDP channels, during end-to-end reporting epochs (which may be greater than or equal to approximately one minute, for example). Such measurements may be used to adapt scheduling policies, congestion control parameters, and overlay QoS policies.

[0168] FIG. 4 illustrates a variety of network applications on a plot of bandwidth sensitivity vs. delay sensitivity. Network applications that are more delay sensitive, such as FPS game position updates, web cache validations, and L4 and T4 signaling, are more likely to be sent on low-latency networks using the disclosed packet forwarding techniques because they either provide a continuous low throughput flow (e.g., below a threshold throughput rate, T), or have short bursts lower than a threshold burst size, S. Similarly, apps with small objects like web browsers are likely to send their initial flows over low latency networks using the disclosed packet forwarding techniques because small HTTP objects and / or initial flows likely small amounts of data (e.g., small bursts below the threshold burst size). The more bandwidth sensitive the app is, the more likely the data is to be forwarded to higher latency links using the disclosed packet forwarding techniques based at least in part on an analysis of the throughput of the flow (e.g., the flow exceeds the threshold throughput rate), with examples of bandwidth sensitive appsincluding, without limitation, VoD chunk downloads, video game maps, emails, game downloads, OS updates, etc.Examples of Network Traffic with TBA-PF2 Schedulers

[0169] FIG. 5 illustrates an example of network traffic with a TBA-PF2 scheduler on a communications network similar to the communications network 200 of FIG. 2. That is, the top graph 500a shows network traffic for a first app object, plot 501 , and a second app object, plot 502, as a function of throughput and time when the disclosed TBA-PF2 policies are applied on a communications network with a high latency GEO link and a low latency LEO link. Similarly, the bottom graph 500b shows network traffic for a third app object, plot 503, as a function of throughput and time when the disclosed TBA-PF2 policies are applied. In each graph 500a, 500b, the shaded portion shows data transmitted with or on the LEO link (e.g., the low latency link) while the unshaded portion shows data transmitted with or on the GEO link (e.g., the high latency link). The solid black line shows the instantaneous throughput at a particular time to show throughput over time.

[0170] The top graph 500a illustrates a first L4 flow (L4 flow-1) with the first app object (app obj-1 ) and the second app object (app obj-2) being sent over the same flow but separated in time. The plot 501 for the first app object experiences a ramping up period followed by a flat period with a decreasing throughput followed by another plateau. The plot 502 of the second app object experiences an abrupt throughput jump which then plateaus until the second app object is sent. The bottom graph 500b illustrates the plot 503 for a second L4 flow (L4 flow-2) with the third app object (app obj-3) having a relatively large burst after a ramping up portion and a flat portion. In these examples, the LEO link helps transmission of the initial slow start and / or the slow start after a long idle time. After the initial slow start (or the slow start after a long idle time), the majority of the data is delivered through the GEO link.

[0171] The graphs 500a, 500b demonstrate how three app objects on two L4 flows are scheduled by TBA-PF2 schedulers on an SD-WAN with LEO and GEO satellite networks (e.g., the communication network 200 of FIG. 2). The graphs 500a, 500b show that up to S bytes of each app object are forwarded to the LEO satellite network and the rest of the data for each app object is forwardedto the GEO satellite network. Graph 500a shows that packets of the first app object are forwarded to the LEO satellite network during a slow start or initial phase where traffic ramps up and then traffic is forwarded to the GEO satellite network once the throughput threshold rate has been reached. This behavior is also seen for the third app object in graph 500b showing the third app object. Graph 500a shows that packets of the second app object are forwarded to the LEO satellite network in the beginning although the first L4 flow did not idle-time-out and thus the transport protocol does not perform a slow-start again. This is due at least in part to the SD-WAN scheduler operating at L3 and not assessing L4 idle-time-out intervals. That is, for the second app object, there is no need to perform the slow start again so the throughput can be set to the peak throughput for the first app object. Graphs 500a, 500b show that packets from a constant bitrate flow under the throughput threshold rate, Tbps, are forwarded to the LEO satellite network until the burst threshold amount, S bytes, has been reached.

[0172] It may be challenging to identify constant low bitrate flows and bursts within a flow, tracking their state, and enforcing TBA-PF2 policies. However, a close approximation can be achieved by adding a Token Bucket Filter (TBF) with a bucket size of S-bytes and token generation rate of T-bps for each 5-tuple flow on the lowest latency network. This method satisfies the TBA-PF2 policies related to forwarding all packets from a constant low bitrate flow under the throughput threshold rate, T bps, and the first S-bytes worth of packets from a flow burst to the lowest latency underlay network. Advantageously, employing a Token Bucket Filter does not require a complex burst detection algorithm as the bucket is refilled with tokens between two bursts.

[0173] FIG. 6 illustrates an example of network traffic with a TBA-PF2 scheduler that employs a token bucket filter on the lowest latency link, the scheduler implemented on a communications network similar to the communications network 200 of FIG. 2. The TBF has a bucket size of S-bytes and a token generation rate of T-bps to approximate the disclosed TBA-PF2 policies.

[0174] The graph 600 shows network traffic for a first app object, plot 601 , and a second app object, plot 602, as a function of throughput and time when the disclosed TBA-PF2 policies are approximated using a TBF as described herein. In the graph 600, the shaded portions show data transmitted with or on the LEO link (e.g., the low latency link) while the unshaded portions show data transmittedwith or on the GEO link (e.g., the high latency link). The solid black lines show the instantaneous throughput at a particular time to show throughput over time. The behavior of the transmission of the first app object, plot 601 , is similar to the first app object of FIG. 5, plot 501, and the behavior of the transmission of the second app object, plot 602, is similar to the second app object of FIG. 5, plot 502.Example implementation of a TBA-PF2 scheduler using TBFs

[0175] FIG. 7 illustrates an example of an implementation of a TBA-PF2 scheduler 700 using token bucket filters. The set of TBFs act as a behavioral classifier for SD-WAN schedulers implementing TBA-PF2 packet forwarding policies. It should be noted that a TBA-PF2 scheduler can be implemented with other methods such as, for example and without limitation, Leaky Buffer Filters (LBF) or simple comparators.

[0176] In a network architecture that has n underlay networks, a TBA-PF2 scheduler can be implemented using n-1 token bucket filters, referred to as a set of TBFs. By convention and to provide superior QoE for time-sensitive applications, TBA-PF2 can assign the link with the lowest latency as link Lo, the highest priority link. The rest of links, Li to Ln-i, can be arranged based on operators’ choices, with a higher link number generally indicating a lower priority. An advantageous approach is to assign the link with the highest capacity and lowest cost as the link with the least priority (Ln-i), though alternate links can be assigned the least priority. The set of TBFs is per-flow based, where a flow is identified as a unique five tuple: (source IP address, source port, destination IP address, destination port, transport protocol). That is, there is a separate set of TBFs assigned to each unique flow.

[0177] The following algorithms are provided using pseudo-code and describe example implementations that provide a behavioral-based SD-WAN scheduler, as described herein. In addition, Table 1 summarizes symbols used in the algorithms.Table 1 - Symbols used to describe algorithms< <

[0178] Algorithm 1 describes an initialization process of the set of n-1 TBFs. TBA-PF2 creates n-1 TBFs for each new flow, f. To help deliver the initial burst, e.g. the slow start phase of TCP flows, TBA-PF2 allocates an initial token size (Bi) that is large enough to accommodate initial burst data, which can be as large as the bucket capacity (Ci). In some implementations, TBA-PF2 creates a set of TBFs for every new flow.>Algorithm 1 : Initialize n - 1 TBFs for a new flow fRequire: n > 0for / = 0 to n - 1 doTi = min(C / , Bi)> end for

[0179] Algorithm 2 describes updating the token for a pre-defined epoch time (Af). For every At period, the SD-WAN controller adds Atx tokens into TBFi until it reaches the maximum bucket capacity (Ci). The rate, , is the same as the desired stable rate which the network operator would assign to every link. For the link Lo with the lowest latency, TBA-PF2 can be configured to assign a rate which is large enough to support time-sensitive applications and to provide a desired QoE.>Algorithm 2: Add new tokens every AtRequi > > >

[0180] Algorithm 3 describes how the TBA-PF2 scheduler chooses the link, Li, to use to transmit a packet, p, upon receiving the packet. When the scheduler receives a new packet, p, it loops through every TBF in order from the highest priority link to the lowest priority link. If the current TBF, TBFi, has enough tokens to send the packet, p, the TBA-PF2 scheduler sends the packet through thecorresponding link Li, provided that the link is uncongested. If the link Li is congested, the packet is sent to the next uncongested underlay link according to the priority order. As discussed herein, a time-sensitive flow with low bandwidth consumption is transmitted over a link with higher priority (e.g., lower latency). On the other hand, a bandwidth-sensitive flow can be transmitted through a link with lower priority (e.g., higher latency). The time complexity and space complexity of Algorithms 1 , 2, and 3 are O(n).Algorithm 3: Receive a packet p for flow fRequire: n > 0 and length(p) > 0for / = 0 to n - 1 doif length(p) < 7} thensend_packet(p, / ') > Send p on un-congested link Li Ti = max(0, 7} - length(p))Returnend ifend forsend_packet(p, n - 1 ) > send p to link Ln-i with the least priority

[0181] FIG. 8 illustrates a flow and logic diagram of a behavior-based traffic classification system implemented in a TBA-PF2 scheduler 800, similar to the TBA-PF2 scheduler 700 of FIG. 7. The TBA-PF2 scheduler 800 includes a single TBF 802 corresponding to an SD-WAN with two underlay networks, similar to the communications network 200 of FIG. 2.

[0182] Different from a DPI-based approach, the behavior-based traffic classification approach does not check the payload of the Internet packet but instead checks the behavior of application flows (e.g., packet length or packet arriving rate). For example, game control flows usually include a series of small packets which carry control signals or target movements, etc. Thus, game control flows only consume a small portion of link capacity.

[0183] Another example is a video streaming application. Typically, bandwidth usage of a video streaming application is dictated by its codec (e.g., 480p YouTube video consumes about 1.5 Mbps capacity). Meanwhile, the client side usually maintains a large playout buffer that buffers about 5-10 seconds of video content to provide better video playout QoE for end users. Once the playoutbuffer is full, the video streaming application has more tolerance for long latency links.

[0184] Responsive to receiving a packet, p, for a particular flow, f, the TBA-PF2 scheduler 800 uses the flow info collector 801 to update statistical information related to the flow, f. The statistical information can include data such as rate, packet length, etc. The TBA-PF2 scheduler 800 then passes the packet, p, to a per flow-based Token Bucket Filter (TBF) 802. For a new flow, f, the TBF 802 allocates an initial token size, Bo, to allow the flow to transmit its initial burst data (see Algorithm 1). The TBF 802 then generates tokens at a rate of n (see Algorithm 2), which corresponds to the desired bitrate allocated to the highest priority link (e.g., the LEO link with low capacity and low latency).

[0185] Upon receiving the packet, p, the TBF 802 tests whether there are sufficient accumulated tokens to send the packet, p, on the high priority link (e.g., the LEO link). If there are sufficient tokens in the bucket, the TBF 802 sends the packet, p, to the high priority link and deducts the size of the packet from the amount of accumulated tokens (see Algorithm 3). If there are not enough tokens in the bucket to transmit the packet, p, on the high priority link or if the corresponding underlay link is congested, the TBF 802 passes the packet, p, to the low priority link (e.g., the GEO link). Implementing such a TBF filter in a TBA-PF2 scheduler, such as the TBA-PF2 scheduler 800, advantageously improves overall link utilization and reduces costs for customers with little or no degradation of QoE.Example Results - Bulk Downloading

[0186] An experimental setup was created to test the described TBA-PF2 technologies. The experimental setup included a Linux machine that supports MP-TCP and that implements HAProxy and that acted as endpoints for multiple GEO and LEO links. The setup implemented two sub-flows for each TCP connection, one sub-flow on the LEO link and the other sub-flow on the GEO link. In the setup, the LEO link has an RTT of 50 ms and the GEO link has an RTT of 600 ms. In this setup, the LEO link is used as the high priority link, Lo, and the GEO link is the low priority link, L-i, with longer latency and / or more economical capacity, which may include a higher capacity.

[0187] FIGS. 9, 10, and 11 illustrate the throughput as a function of time for different underlay networks in a bulk download test. The bulk downloadrepresented a throughput-sensitive application (an emulated HD video flow) to test the behavior or the experimental setup. FIG. 9 illustrates a graph 900 of LEO subflow bandwidth usage 901 as a function of time for bulk downloading using the experimental setup. FIG. 10 illustrates a graph 1000 of GEO sub-flow bandwidth usage 1001 as a function of time for bulk downloading using the experimental setup. FIG. 11 illustrates a graph 1100 of the aggregated bandwidth usage 1101 of the SD-WAN for the bulk download using the experimental setup.

[0188] FIG. 9 demonstrates that because a large initial token (Bo = 10 MB) was granted on the low-capacity, low-latency LEO link, the initial data was transmitted through the LEO link at high speed (~2.5 Mbps). After the initial burst, the throughput of the LEO sub-flow reduced to around 300 kbps at about the 8th second. The reason for the throughput reduction is that the token generation rate was set to 300 kbps for the LEO link. After the initial burst consumed all of the tokens in the bucket, the LEO sub-flow was throttled to save the bandwidth of the LEO link for other delay-sensitive flows.

[0189] FIG. 10 demonstrates the bandwidth usage of the TCP sub-flow over the long-RTT, high-bandwidth GEO link. Because the initial burst was sent through the low RTT link, the transmission throughput ramps up relatively quickly. After the 10th second, the packet arrival rate at the TBA-PF2 scheduler was faster than the token generation rate, ro, for the LEO link. In this case, after the accumulated tokens, To, were consumed, the TBA-PF2 scheduler could only send the data packets to the lower priority GEO link. The throughput of the sub-flow over the GEO link reaches about 100 Mbps and maintains a stable rate at about 125 Mbps, which is higher than the data rate (~300 kbps) over the LEO link. This demonstrates that the bandwidth-sensitive traffic burst was successfully shifted from the high-priority LEO link to the low-cost GEO link with little or no degradation in performance (e.g., as indicated by throughput).

[0190] FIG. 11 demonstrates the aggregated throughput over the SD-WAN network (including the LEO link and the GEO link). Due to the implementation of the TBA-PF2 policies described herein, the majority of the bandwidth-sensitive traffic was transmitted through the low-cost GEO link. Moreover, even though the GEO link was fully utilized, the high-priority LEO link still had at least about 80% of its available bandwidth which could be used to transmit other delay-sensitive or mission-critical traffic.Example Results - Video Streaming

[0191] The same experimental setup was used to test video streaming performance. FIG. 12 illustrates a graph 1200a of the throughput of the video traffic with the GEO sub-flow (black bars) and the LEO sub-flow (white bars) and a graph 1200b of the throughput on just the LEO link (white bars). As can be seen from the graphs 1200a, 1200b, the majority of the bytes from the bursts of video streaming were delivered through the high-latency, large-capacity GEO link. The video streaming application initially sent a big chunk of video data to quickly fill up the media buffer, as indicated in graph 1200b. After the initial burst, the video traffic was sent as groups of bursts over the GEO link. The graph 1200b shows the behavior of the high-priority, low-latency LEO link. Similar to the bulk downloading case, after the initial burst, the throughput through the LEO link was paced to be a TBF-style rate control to allow relatively small bursts to go through the LEO link. Since video data is typically sent in chunks, small bursts can be sent over the LEO link. The idle gaps between the bursts are available for use by other applications.

[0192] During testing, multiple 4K HD videos were streamed through an SD-WAN with a TBA-PF2 scheduler, the SD-WAN including a 3 Mbps LEO link and a 150 Mbps GEO link. The TBF bucket size was set to 10 Mbytes and the token generation rate was set to 500 kbps. Simultaneously, a user was surfing web pages or playing role-playing games (RPG) or first person shooter (FPS) games. The web browsing and video games are delay sensitive applications which are usually measured by page load time or server response time. Even with four or five 4K HD videos running in the background, the web browsing experience was close to that of a low-latency broadband network and the users playing RPG and FPS games noticed little or no degradation in performance. The HD videos streamed smoothly without any resolution reduction or rebuffering. This demonstrates that the disclosed TBA-PF2 technologies improve the QoE for time-sensitive applications with little or no degradation in QoE for bandwidth-sensitive applications.Example delay sensitivity classifier for TBA-PF2

[0193] In some implementations, a delay sensitivity classifier (DSC) can be implemented in conjunction with the TBA-PF2 technologies disclosed herein. The DSC can be configured to detect 5-tuple bursts and to set a delay-sensitivitylevel for each incoming packet to increase utilization of the network. The delaysensitivity level can be used along with an assessment of network utilization to route packets, where the network utilization can be a function of customer experience and operational costs. Thus, network utilization can account for superior bandwidth economics (e.g., cost-per-bit) in determining scheduling and packet routing. In some implementations, the delay-sensitivity level can be expressed using information in headers (e.g., IP headers), such as a differentiated service code point or DSCP (a bit long field in the IP header), or an integer attached with a socket buffer (e.g., a socket buffer mark or skb-mark) and / or other methods for including this information in packet metadata. In some implementations, DSCP can be used to classify, mark, and manage traffic to ensure targeted QoE and can be used to carry delay sensitivity levels. In some implementations, skb-mark can be used to mark packets to enable differentiated handling of packets based on delay sensitivity levels. The DSC can be implemented as an ingress module on an external-facing network interface. Thus, the DSC can be configured to control utilization of capacity through the low-latency network based on the assigned delay-sensitivity classifier.

[0194] FIG. 13 illustrates an example DSC 1310 for use with TBA-PF2 packet forwarding technologies implemented in an SD-WAN scheduler 1355 in an SD-WAN 1300 that includes a plurality of underlay networks 1340 that include a LTE / LEO link 1340a, a MEO link 1340b, and a GEO link 1340c. The DSC 1310 is configured to receive a packet, p, and to detect 5-tuple bursts and to set a delaysensitivity level, P— Li, based on the packet. The delay-sensitivity level can be set to correspond to a particular link of the plurality of underlay networks 1340. For example, a delay-sensitivity level P-Lo indicates to the SD-WAN scheduler 1355 that the packet should be sent to the highest priority link, a delay-sensitivity level P-Li indicates to the SD-WAN scheduler 1355 that the packet should be sent to the middle priority link, and a delay-sensitivity level P-L2 indicates to the SD-WAN scheduler 1355 that the packet should be sent to the lowest priority link.

[0195] The SD-WAN scheduler 1355 receives the delay-sensitivity level from the DSC 1310 and directs packets based at least in part on the received delaysensitivity level. In some implementations, the highest priority packets (P-Lo) go to the active highest priority link irrespective of its congested state. If the LTE / LEO link 1340a is active, then the packet with the sensitivity level of P-Lo is sent to theLTE / LEO link 1340a and if the LTE / LEO link 1340a is inactive, the packet with the sensitivity level of P-Lo is sent to the MEO link 1340b.

[0196] The SD-WAN scheduler 1355 is configured to send packets go to the link that matches the priority indicated by the sensitivity level determined by the DSC 1310. However, the SD-WAN scheduler 1355 can send packets to the link that is of a lower priority indicated by the sensitivity level if the matching priority link is congested. That is, if the matching link is congested, the SD-WAN scheduler 1355 sends the packet to the next priority link that is uncongested. For example, if the MEO link 1340b is not congested, then P-Li is sent to the MEO link; if the MEO link 1340b is congested, then P-Li is sent to the GEO link 1340c; if the GEO link 1340c is also congested, then P-Li is sent to the LTE / LEO link 1340a. As another example, if the GEO link 1340c is congested, then P-L2 is sent to the MEO link 1340b; if the MEO link 1340b is also congested, then P-L2 is sent to the LTE / LEO link 1340a. In such implementations, the priorities of the link are as follows: the LTE / LEO link 1340a is priority Lo and L4, MEO link 1340b is priority Li and L3, and the GEO link 1340c is L2.

[0197] When all links are congested, the SD-WAN scheduler is configured to forward a packet to the matching or lower priority active link. If the matching priority link is inactive, the packet goes to the next priority active link. If all links are active, P-Lo is sent to the LTE / LEO link 1340a, P-L1 is sent to the MEO link 1340b, and P-L2 is sent to the GEO link 1340c. If the LTE / LEO link 1340a is inactive, P-Lo and P-L1 are sent to the MEO link 1340b and P-L2 is sent to the GEO link 1340c. If the MEO link 1340b is inactive, P-Lo is sent to the LTE / LEO link 1340a and P-L1 and P-L2 are sent to the GEO link 1340c. If the GEO link 1340c is inactive, P-Lo is sent to the LTE / LEO link 1340a and P-L1 and P-L2 are sent to the MEO link 1340b.

[0198] The congestion state update interval of the links are configured to be less than the RTT of the associated link. In some implementations, the congestion state of a link can be updated when each protocol data unit (PDU) ack is received at the sender side. In some implementations, the PDU acknowledgement for a long-latency link can be sent to a low-latency link to reduce the effective RTT. Each underlay link can be configured to provide an independent, configurable parameter to enable and disable small PDU forwarding without an additional delay when a packet is forwarded to the link.

[0199] FIG. 14 illustrates an example implementation of a DSC 1400 for a TBA-PF2 packet scheduler or SD-WAN scheduler, examples of which are described herein. The DSC 1400 implements a per-flow, two-rate, and three-color meter for determining delay-sensitivity. First, the DSC 1400 determines the 5-tuple of the incoming packet and the direction of the incoming packet. If the packet is new, a new flow is created and the DSC 1400 determines a color, or priority, of the flow. In some implementations, once the color or priority of the flow is determined, the flow is not re-classified by the DSC 1400, or the flow retains the assigned color or priority for the lifetime of the flow. The DSC 1400 can be implemented with a different number of rates and a different number of colors, the two-rate and three-color meter implementation is provided merely as an example. The delaysensitivity level process performed by the DSC 1400 is configured to provide results similar to those presented herein with respect to FIG. 6.

[0200] The DSC 1400 includes a committed bucket 1404 (where CBS refers to the committed burst size and CIR refers to the committed information rate) and a peak bucket 1406 (where PBS refers to the peak burst size and PIR refers to the peak information rate). The committed bucket 1404 is refilled using a rate corresponding to the CIR and the peak bucket 1406 is refilled using a rate corresponding to the PIR. Thus, the DSC 1400 implements a two-rate method for determining delay sensitivity, or there are two bucket rates utilized by the DSC 1400 in the process of determining delay sensitivity of a flow. The amount of tokens in the peak bucket 1406 is configured to be greater than or equal to the amount of tokens in the committed bucket 1404.

[0201] At block 1402, the DSC 1400 receives an incoming packet with a length of B. At block 1410, the DSC 1400 compares the packet length, B, to the amount of committed tokens, Tc, in the committed bucket 1404. If the packet length is larger than the amount of tokens in the committed bucket 1404, the amount of tokens in the committed bucket is set to 0 in block 1420. Then, at block 1430, the DSC 1400 compares the packet length, B, to the amount of burst tokens, Te, in the peak bucket 1406. If the packet length is larger than the amount of tokens in the peak bucket 1406, the amount of burst tokens in the peak bucket is set to 0 in block 1440. The DSC 1400 then sets the priority or color of the packet to RED in block 1441 , corresponding to a priority of P-L2.

[0202] If, in block 1410, the packet length is less than the amount of tokens in the committed bucket, the amount of committed tokens, Tc, in the committed bucket 1404 is reduced by the packet length, B, in block 1411 and the amount of peak tokens, Te, in the committed bucket 1404 is reduced by the packet length, B, in block 1412. The color of the packet is set to GREEN in block 1413, corresponding to a priority of P-Lo.

[0203] If, in block 1430, the packet length is less than the amount of tokens in the peak bucket, the amount of peak tokens, Te, in the peak bucket 1406 is reduced by the packet length, B, in block 1431. The color of the packet is set to YELLOW in block 1432, corresponding to a priority of P-Li.Example SD-WAN Architecture

[0204] FIG. 15 illustrates a block diagram of a communication network 1500 implementing a SD-WAN design that includes a CPE proxy 1510 and an ISP proxy 1550 with SD-WAN packet forwarding technologies. The SD-WAN packet forwarding technologies can implement TBA-PF2 packet forwarding policies, as described herein. The SD-WAN packet forwarding technologies can be implemented within the CPE proxy 1510 and / or the ISP proxy 1550 or as part of the SD-WAN topology management components 1522, 1542.

[0205] The communications network 1500 includes an SD-WAN architecture that utilizes a LEO satellite network 1505 and a GEO satellite network 1506 as underlay networks to communicate with a core network 1507. Each underlay network has a corresponding underlay network interface 1501, 1502, 1509. The core network 1507 also includes an L3 LB switch 1508 to manage communication between the ISP proxy 1550 and the core network 1507. The SD-WAN topology management components 1522, 1542 can manage operation of the respective underlay L3 network interfaces 1501, 1502, 1509. Overlay QoS policies 1521, 1541 can also be implemented to manage network resources across the communications network 1500.

[0206] The communication network 1500 provides communication between a customer network 1518 and an external network, such as the Internet 1560. The customer network 1518 can connect to the CPE proxy 1510 through a security module 1517, which can be implemented as a user terminal or a mobileterminal. The ISP proxy 1550 connects to the Internet 1560 through a gateway 1557, as described herein.

[0207] The CPE proxy 1510 includes an L4 UDP session module 1511, an unreliable MP-QUIC session module 1512, L4 QUIC connection modules 1513, 1514, a reliable MP-QUIC session module 1515, and an L4 TCP termination module 1516. Similarly, the ISP proxy 1550 includes an L4 UDP session module 1551, an unreliable MP-QUIC session module 1552, L4 QUIC connection modules 1553, 1554, a reliable MP-QUIC session module 1555, and an L4 TCP termination module 1556. The SD-WAN architecture implements MP-QUIC for tunneling and communication through the SD-WAN overlay.

[0208] The communications network 1500 implements an L3 SD-WAN packet forwarder for UDP traffic flows. Outer UDP session modules 1511, 1551 are mapped to the unreliable inner MP-QUIC session modules 1512, 1552 while inner MP-QUIC session modules 1515, 1555 perform opportunistic retransmission on loss detection.

[0209] The communications network 1500 implements an L4 SD-WAN Proxy for TCP. Outer TCP sessions are terminated at the L4 TCP termination modules 1516, 1556 and mapped to reliable inner MP-QUIC session modules 1515, 1555, which provides advantages similar to performance enhancing proxies in GEO satellite networks and may particularly advantageous when no low-latency network is available so that the slow-start algorithm can be bypassed.

[0210] The communications network 1500 implements L4 underlay (QUIC) connections and performs end-to-end congestion control while tracking capacity availability. The capacity availability can be provided to a SD-WAN packet forwarding scheduler, examples of which are described herein. The SD-WAN packet forwarding scheduler is configured to forward packets from MP-QUIC sessions to QUIC connections to achieve various goals. As described herein, delay-sensitive packet classification is advantageous for wireless SD-WAN architectures with heterogeneous latencies to achieve QoE-optimized capacity arbitrage.Examples of Network Traffic with DSC-F Schedulers

[0211] FIG. 16 illustrates examples of network traffic with a DSC-F scheduler on a communications network similar to the communications network300 of FIG. 3. That is, the top graph 1600a shows network traffic for a first app object, plot 1601, and a second app object, plot 1602, as a function of throughput and time when the disclosed DSC-F policies are applied on a communications network with a high latency GEO link and a low latency LEO link. Similarly, the middle graph 1600b shows network traffic for a third app object, plot 1603, as a function of throughput and time when the disclosed DSC-F policies are applied. Similarly, the bottom graph 1600c shows network traffic for another L4 flow, plot 1604, as a function of throughput and time when the disclosed DSC-F policies are applied. In each graph 1600a, 1600b, 1600c, the shaded portion shows data transmitted with or on the LEO link (e.g., the low latency link) while the unshaded portion shows data transmitted with or on the GEO link (e.g., the high latency link). The solid black line shows the instantaneous throughput at a particular time to show throughput over time.

[0212] The top graph 1600a illustrates a first L4 flow with the first app object and the second app object being sent over the same flow but separated in time. The first app object experiences a ramping up period followed by a flat period with a decreasing throughput followed by another plateau. The second app object experiences an abrupt throughput jump which then plateaus until the second app object is sent. The middle graph 1600b illustrates a second L4 flow with the third app object having a relatively large burst after a ramping up portion and a flat portion. In these examples, the LEO link helps transmission of the initial N-bytes or N-packets of the burst with the remaining bytes or packets sent over the GEO link. After the initial N-bytes or N-packets, the majority of the data is delivered through the GEO link. This demonstrates the DSC-F policies corresponding to bursts: identify a 5-tuple burst and forward the beginning of the burst to the LEO link. The bottom graph 1600c illustrates a third L4 flow that represents a continuous low throughput stream (which can include an ack stream, VoIP data, FPS position updates, and the like). For such streams, the DSC-F policies are configured to forward the packets to the LEO link rather than using the GEO link for the data.

[0213] The graphs 1600a, 1600b, 1600c demonstrate how app objects and L4 flows are scheduled by DSC-F schedulers on an SD-WAN with LEO and GEO satellite networks (e.g., the communication network 300 of FIG. 3). The graphs 1600a, 1600b, 1600c show that delay sensitive data is forwarded to the LEO satellite network and the rest of the data for each app object is forwarded tothe GEO satellite network. Graph 1600c shows that packets from a constant bitrate flow under a throughput threshold rate are marked as delay sensitive and forwarded to the LEO satellite network.

[0214] In some implementations, the disclosed DSC and TBF algorithms can be used to identify bursts within a flow and / or constant low bitrate flows. Other methods or algorithms may also be employed to identify delay sensitive packets to inform the DSC-F schedulers.Example SD-WAN Architecture for Hybrid Networks

[0215] FIG. 17 illustrates a block diagram of a communication network 1700 implementing a SD-WAN architecture for hybrid networks, similar to the communication network 1500 of FIG. 15. The communication network includes an SEEP 1710 and an SDEP 1750 with SD-WAN packet forwarding technologies. The SD-WAN packet forwarding technologies can implement DSC-F packet forwarding policies, as described herein. The SD-WAN packet forwarding technologies can be implemented within the SEEP 1710 and / or the SDEP 1750.

[0216] The communications network 1700 includes an SD-WAN architecture that utilizes a plurality of access networks 1781, 1782 and a core network 1783 as underlay networks to communicate between the SEEP 1710 and the SDEP 1750. The SEEP 1710 includes a plurality of bearers, e.g., bearer 1 1713 and bearer 2 1714 where a bearer can be bonded to an access network or network path. In some implementations, a plurality of bearers can be bonded to a single access network. Similarly, the SDEP 1750 includes a plurality of bearers, e.g., bearer 1 1753 and bearer 2 1754 corresponding to the bearer 1 1713 of SEEP 1710 and the bearer 2 1714 of SEEP 1710, respectively.

[0217] The communication network 1700 provides communication between a customer network 1718 and an external network, such as the Internet 1760. The customer network 1718 can connect to the SEEP 1710 through a security module 1717, which can be implemented as a user terminal or a mobile terminal. The SDEP 1750 connects to the Internet 1760 through a gateway 1757, as described herein.

[0218] The SEEP 1710 includes overlay UDP session modules 1711, unreliable channels 1712, reliable channels 1715, and overlay TCP termination modules 1716 with a delay sensitive scheduler 1719 forwarding packets from theunreliable channels 1712 and the reliable channels 1715 to particular bearer networks (e.g., bearer 1 or bearer 2). Similarly, the SDEP 1750 includes overlay UDP session modules 1751, unreliable channels 1752, reliable channels 1755, and overlay TCP termination modules 1756 with a delay sensitive scheduler 1759 forwarding packets from the unreliable channels 1752 and the reliable channels 1755 to particular bearer networks (e.g., bearer 1 or bearer 2). The SD-WAN architecture implements MP-QUIC for end-to-end tunneling and communication through the SD-WAN overlay.

[0219] The communication network 1700 can be configured to perform end-to-end congestion control while tracking capacity availability. The capacity availability can be provided to a SD-WAN packet forwarding scheduler, examples of which are described herein. The SD-WAN packet forwarding scheduler is configured to forward packets from MP-QUIC sessions to QUIC connections to achieve various goals. As described herein, delay-sensitive packet classification is advantageous for wireless SD-WAN architectures with heterogeneous latencies to achieve QoE-optimized capacity arbitrage.

[0220] In operation, the scheduler 1719 at the SEEP 1710 and the corresponding scheduler 1759 at the SDEP 1750 perform packet-to-bearer mapping according to a plurality of scheduling rounds, as further described with respect to FIG. 18. In a delay sensitivity round, channel data frames corresponding to packets classified as delay-sensitive by the DSC-F policy are preferentially scheduled onto one or more low-latency bearers. Delay-sensitive traffic may include, for example, the first N-bytes or first N-packets of a newly detected 5-tuple burst, packets belonging to newly established QUIC 5-tuple flows, overlay transport acknowledgment streams, and continuous low-throughput delay-sensitive flows. By forwarding such traffic over low-latency bearers during the delay sensitivity round, the schedulers reduce control loop delay, accelerate congestion window growth for overlay transport protocols, and improve user-perceived responsiveness for small objects and interactive applications.

[0221] Following the delay sensitivity round, the scheduler performs a throughput sensitivity round in which channel data frames classified as throughputsensitive are scheduled onto higher-latency bearers, that may also have superior bandwidth economics, until such bearers reach a congestion-controlled operating limit. This round enables economical utilization of bonded underlay capacity whilemaintaining separation between delay-sensitive and throughput-sensitive traffic. During this round, the bearer congestion control loops (e.g., B1 and B2 congestion control loops shown in FIG. 17) regulate transmission rates to avoid packet loss and buffer bloat at the corresponding access networks.

[0222] After the throughput sensitivity round, a max bonded round is performed in which remaining or overflow throughput-sensitive channel data frames are scheduled onto uncongested low-latency bearers when available. This overflow behavior ensures that total bonded underlay capacity is maximally utilized when traffic demand exceeds the capacity of higher-latency bearers, while still prioritizing delay-sensitive traffic in earlier rounds. In hybrid environments having bearers with substantially different RTTs, the round-based scheduling structure reduces unnecessary underlay switching for individual channels and avoids spreading data frames of a single UDP channel across bearers with widely divergent RTTs, thereby minimizing overlay packet delivery jitter.

[0223] By organizing scheduling decisions into delay sensitivity, throughput sensitivity, and max bonded rounds, the architecture of FIG. 17 provides coordinated interaction between channel classification, bearer congestion control, and bonded capacity utilization. This structure enables the SEEP 1710 and the SDEP 1750 to simultaneously (i) preserve low control loop delay for delaysensitive traffic, (ii) economically utilize longer latency underlay networks for throughput-sensitive traffic, and (iii) adapt dynamically to congestion and capacity fluctuations across heterogeneous access networks.

[0224] In certain implementations, the schedulers 1719 and 1759 operate as supervisory control elements layered above independent bearer congestion control loops. The scheduling rounds function as an outer control layer that selects bearer assignment for channel data frames, while the bearer congestion control loops function as inner control layers regulating transmission rate for each underlay path. By coordinating round-based scheduling with live congestion feedback from each bearer, oscillatory behavior can be inhibited or prevented that can arise in hybrid RTT environments when traffic is shifted between paths without regard to congestion state. This layered control structure improves stability, reduces jitter, and enhances responsiveness relative to schedulers that perform packet assignment without integrating bearer congestion control state.Example DSC-F Method

[0225] FIG. 18 illustrates a flow chart of an example method 1800 for performing delay sensitive classification and forwarding. The method 1800 can be performed by any of the schedulers described herein. For ease of description, the method 1800 is described as being performed by a scheduler, but it is to be understood that any step of the method 1800 can be performed by a subcomponent and / or a collection of components of the SD-WAN architectures disclosed herein.

[0226] In block 1802, the scheduler performs a delay sensitivity round wherein the scheduler forwards delay sensitive packets and / or delay sensitive payloads to a low latency bearer network. In some implementations, fairness among channels within the round is controlled using channel priority and / or weight. In general, congestion is not expected to occur in the delay sensitivity round since delay sensitive traffic is configured to be small in volume. For example, flows in this round are typically overlay transport acks, small 5-tuple bursts, etc.

[0227] During the delay sensitivity round, the scheduler can identify a 5-tuple burst and forwards the beginning of the burst to the low latency network bearer (e.g., a LEO satellite network). For example, the scheduler marks the first N-bytes or the first N-packets as delay sensitive to be forwarded to the low latency bearer network. Similarly, during this round the scheduler can identify a continuous low-throughput 5-tuple flow stream and forwards the packets to the low latency network bearer. For example, the scheduler can be configured to forward packets from a 5-tuple flow stream with a rate less than a threshold rate (e.g., M-kbps) to a low latency bearer network. This advantageously results in low latency for low-bitrate applications. Similarly, the scheduler during this round can be configured to forward SD-WAN data transmission ack streams to the low latency bearer network. This can improve or optimize overlay transport protocol control loop delays. Similarly, the scheduler during this round can identify small HTTP objects as delay sensitive and forward the corresponding packets to the low latency bearer network. This advantageously provides a responsive web experience to a user.

[0228] In block 1804, the scheduler performs a throughput sensitivity round wherein the scheduler forwards throughput sensitive packets and / or throughput sensitive payloads to the economical higher latency network bearer until that bearer is full. In some implementations, fairness among channels withinthe round is controlled using channel priority and / or weight. In some implementations, any packet that is not marked or identified as delay sensitive in block 1802 is marked as throughput sensitive in block 1802.

[0229] In block 1806, the scheduler performs a max bonded round wherein the scheduler forwards leftover or overflow throughput sensitive packets after the throughput sensitive round to a low latency network bearer to maximally utilize bonded capacity utilization. In some implementations, fairness among channels within the round is controlled using channel priority and / or weight.

[0230] The round-based DSC-F scheduling method illustrated in FIG. 18 provides enhanced QoE for both end-to-end UDP and TCP flows and for application-layer performance. By distinguishing delay-sensitive and throughputsensitive traffic and scheduling packets accordingly, the system improves transport responsiveness while maintaining efficient bonded capacity utilization.

[0231] For encrypted UDP-based transports such as QUIC, DSC-F reduces initial transport and security protocol negotiation delay by forwarding connection establishment packets, cryptographic handshake packets, and early control packets over low-latency underlay networks during the delay sensitivity round. This reduces time-to-first-byte and shortens session setup latency. During the initial slow-start phase of a QUIC flow, forwarding early data packets over a low-latency bearer accelerates growth of the QUIC congestion window, enabling the overlay transport to more rapidly utilize available underlay capacity. Additionally, DSC-F forwards QUIC acknowledgment packets over low-latency bearers so that control feedback loops are shortened in hybrid high- and low-latency environments. By reducing control loop delay associated with high-latency underlay paths, the effective transmission window can grow more predictably and with reduced oscillation.

[0232] For end-to-end TCP sessions, where TCP termination is performed at SEEP and SDEP, DSC-F improves performance by forwarding SD-WAN reliable-channel acknowledgment packets over low-latency underlay bearers. Because reliable SD-WAN channels replace the original end-to-end TCP congestion control loop across heterogeneous underlay paths, control feedback associated with the SD-WAN bearer congestion control loop can be selectively routed over lower-latency networks. In a hybrid GEO-LEO SD-WAN environment, this routing behavior can reduce effective transmission control loop delayassociated with the high-latency GEO bearer by nearly half, thereby improving goodput and reducing recovery time following congestion events.

[0233] DSC-F further improves or optimizes delivery of small HTTP objects and similar short-lived transfers. By forwarding the first N-bytes or N-packets of each TCP or UDP burst to a low-latency bearer, DSC-F reduces completion time for small objects. Because web traffic commonly exhibits a heavytail object size distribution in which most objects are small, prioritizing the initial portion of such bursts significantly improves user-perceived web responsiveness during page navigation. Similarly, during streaming video startup, forwarding early burst packets over a low-latency bearer enables more rapid filling of the client playback buffer, reducing startup delay and improving perceived streaming quality.

[0234] Through these mechanisms, the DSC-F method enhances end-to-end transport efficiency, reduces effective control loop delay in hybrid RTT environments, and improves user-perceived responsiveness across interactive, transactional, and streaming applications while maintaining efficient utilization of bonded heterogeneous underlay networks.Example SD-WAN Scheduler

[0235] FIG. 19 illustrates a block diagram of a scheduler 770 of a communications system, such as a SD-WAN scheduler, examples of which are described herein with reference to FIGS. 1, 2, 3, 7, 8, 13, 15, and 17. The scheduler 770 is configured to implement DSC-F and / or TBA-PF2 packet forwarding policies to provide QoE-optimized capacity arbitrage, as described herein. The scheduler 770 can employ any method described herein for packet forwarding, such as the example methods described herein with reference to FIGS. 7, 14, and 18.

[0236] The scheduler 770 can include hardware, software, and / or firmware components for scheduling network traffic in a hybrid SD-WAN architecture. The scheduler 770 includes a data store 771 , one or more processors 773, one or more network interfaces 775, a connection module 772, a throughput module 774, and a burst module 776. Components of the scheduler 770 can communicate with one another, with external systems, and with other components of a network using communication bus 779. The scheduler 770 can be implemented using one or more computing devices. For example, the scheduler 770 can be implemented using a single computing device, multiple computingdevices, a distributed computing environment, or it can be located in a virtual device residing in a public or private computing cloud. In a distributed computing environment, one or more computing devices can be configured to provide the modules 772, 774, 776 to provide the described functionality.

[0237] The scheduler 770 includes the connection module 772 to determine when a new flow or a timed-out-connection flow is initiated. In such instances, packets are forwarded to the lowest latency underlay network that is not congested. Similarly, the connection module 772 can be configured to determine overlay ack streams to be forwarded to the lowest latency underlay network. The scheduler 770 includes the throughput module 774 to determine whether a flow is using a throughput that is less than a threshold throughput rate, T-bps. In such instances, packets are forwarded to the lowest latency underlay network that is not congested. This can include low throughput streams, as described herein. The scheduler includes the burst module 776 to determine whether a burst size of a flow is less than a threshold burst size, S-bytes. In such instances, packets are forwarded to the lowest latency underlay network that is not congested. Otherwise, the packets are forwarded to an underlay network with sufficient available capacity based at least in part on economics. In some implementations, the lowest latency network can be used for bursts that exceed the threshold depending at least in part on the cost-per-bit economics. In some implementations, the burst module 776 is configured to identify traffic that includes small HTTP objects to be marked as delay sensitive. In some implementations, the burst module 776 is configured to forward the first N-packets or N-bytes to the lowest latency underlay network, as described herein.

[0238] The scheduler 770 includes one or more processors 773 that are configured to control operation of the modules 772, 774, 776 and the data store 771. The one or more processors 773 implement and utilize the software modules, hardware components, and / or firmware elements configured to perform packet forwarding functions described herein. The one or more processors 773 can include any suitable computer processors, application-specific integrated circuits (ASICs), field programmable gate array (FPGAs), or other suitable microprocessors. The one or more processors 773 can include other computing components configured to interface with the various modules and data stores of the scheduler 770.

[0239] The scheduler 770 includes the data store 771 configured to store configuration data, congestion control algorithms, capacity statuses, network statuses, network characteristics and capabilities, control commands, databases, algorithms, executable instructions (e.g., instructions for the one or more processors 773), and the like. The data store 771 can be any suitable data storage device or combination of devices that include, for example and without limitation, random access memory, read-only memory, solid-state disks, hard drives, flash drives, bubble memory, and the like. The data store 771 can be a non-transitory computer-readable medium. The data store 771 can store processor-executable instructions to implement one or more of the modules 772, 774, 776 and / or any disclosed methods.Additional Embodiments and Terminology

[0240] As used herein, the quality of experience, or QoE, can refer to the customer’s perceptions of a service provider’s network based on the customer’s experience using the service provider’s network. The QoE is influenced by the customer’s expectations and experiences which, in a network, relate to the expected network speed (e.g., the bandwidth offered, advertised, or promised by the service provider) and the experienced network speed (e.g., the bandwidth actually experienced by the customer).

[0241] As used herein, the term token can be used to represent a unit of bytes of a predetermined size useful in token bucket filters or token bucket algorithms. As used herein, the term bucket can refer to a data structure that can hold a certain number of tokens. As used herein, token bucket algorithms can refer to algorithms that inspect buckets to see there are sufficient tokens to transmit data on the corresponding link. If there are sufficient tokens, the appropriate number of tokens are removed (or cashed out) and the packet is passed for transmission. The bucket can be refilled at a refill rate until the bucket is full.

[0242] Although the technologies are described herein primarily in relation to SD-WAN architectures, it is to be understood that the disclosed technologies can also be used in relation to SD-LAN architectures. Moreover, the disclosed technologies can be used with other similar architectures that use multiple wireless underlay networks to provide a targeted QoE for subscribers or users.

[0243] The present disclosure describes various features, no single one of which is solely responsible for the benefits described herein. It will be understood that various features described herein may be combined, modified, or omitted, as would be apparent to one of ordinary skill. Other combinations and subcombinations than those specifically described herein will be apparent to one of ordinary skill, and are intended to form a part of this disclosure. Various methods are described herein in connection with various flowchart steps and / or phases. It will be understood that in many cases, certain steps and / or phases may be combined together such that multiple steps and / or phases shown in the flowcharts can be performed as a single step and / or phase. Also, certain steps and / or phases can be broken into additional sub-components to be performed separately. In some instances, the order of the steps and / or phases can be rearranged and certain steps and / or phases may be omitted entirely. Also, the methods described herein are to be understood to be open-ended, such that additional steps and / or phases to those shown and described herein can also be performed.

[0244] Some aspects of the systems and methods described herein can advantageously be implemented using, for example, computer software, hardware, firmware, or any combination of computer software, hardware, and firmware. Computer software can comprise computer executable code stored in a computer readable medium (e.g., non-transitory computer readable medium) that, when executed, performs the functions described herein. In some embodiments, computer-executable code is executed by one or more general purpose computer processors. A skilled artisan will appreciate, in light of this disclosure, that any feature or function that can be implemented using software to be executed on a general purpose computer can also be implemented using a different combination of hardware, software, or firmware. For example, such a module can be implemented completely in hardware using a combination of integrated circuits. Alternatively or additionally, such a feature or function can be implemented completely or partially using specialized computers designed to perform the particular functions described herein rather than by general purpose computers.

[0245] Multiple distributed computing devices can be substituted for any one computing device described herein. In such distributed embodiments, the functions of the one computing device are distributed (e.g., over a network) such that some functions are performed on each of the distributed computing devices.

[0246] Some embodiments may be described with reference to equations, algorithms, and / or flowchart illustrations. These methods may be implemented using computer program instructions executable on one or more computers. These methods may also be implemented as computer program products either separately, or as a component of an apparatus or system. In this regard, each equation, algorithm, block, or step of a flowchart, and combinations thereof, may be implemented by hardware, firmware, and / or software including one or more computer program instructions embodied in computer-readable program code logic. As will be appreciated, any such computer program instructions may be loaded onto one or more computers, including without limitation a general purpose computer or special purpose computer, or other programmable processing apparatus to produce a machine, such that the computer program instructions which execute on the computer(s) or other programmable processing device(s) implement the functions specified in the equations, algorithms, and / or flowcharts. It will also be understood that each equation, algorithm, and / or block in flowchart illustrations, and combinations thereof, may be implemented by special purpose hardware-based computer systems which perform the specified functions or steps, or combinations of special purpose hardware and computer-readable program code logic means.

[0247] Furthermore, computer program instructions, such as embodied in computer-readable program code logic, may also be stored in a computer readable memory (e.g., a non-transitory computer readable medium) that can direct one or more computers or other programmable processing devices to function in a particular manner, such that the instructions stored in the computer-readable memory implement the function(s) specified in the block(s) of the flowchart(s). The computer program instructions may also be loaded onto one or more computers or other programmable computing devices to cause a series of operational steps to be performed on the one or more computers or other programmable computing devices to produce a computer-implemented process such that the instructions which execute on the computer or other programmable processing apparatus provide steps for implementing the functions specified in the equation(s), algorithm(s), and / or block(s) of the flowchart(s).

[0248] Some or all of the methods and tasks described herein may be performed and fully automated by a computer system. The computer system may,in some cases, include multiple distinct computers or computing devices (e.g., physical servers, workstations, storage arrays, etc.) that communicate and interoperate over a network to perform the described functions. Each such computing device typically includes a processor (or multiple processors) that executes program instructions or modules stored in a memory or other non-transitory computer-readable storage medium or device. The various functions disclosed herein may be embodied in such program instructions, although some or all of the disclosed functions may alternatively be implemented in applicationspecific circuitry (e.g., ASICs or FPGAs) of the computer system. Where the computer system includes multiple computing devices, these devices may, but need not, be co-located. The results of the disclosed methods and tasks may be persistently stored by transforming physical storage devices, such as solid state memory chips and / or magnetic disks, into a different state.

[0249] Unless the context clearly requires otherwise, throughout the description and the claims, the words “comprise,” “comprising,” and the like are to be construed in an inclusive sense, as opposed to an exclusive or exhaustive sense; that is to say, in the sense of “including, but not limited to.” The word “coupled”, as generally used herein, refers to two or more elements that may be either directly connected, or connected by way of one or more intermediate elements. Additionally, the words “herein,” “above,” “below,” and words of similar import, when used in this application, shall refer to this application as a whole and not to any particular portions of this application. Where the context permits, words in the above Detailed Description using the singular or plural number may also include the plural or singular number respectively. The word “or” in reference to a list of two or more items, that word covers all of the following interpretations of the word: any of the items in the list, all of the items in the list, and any combination of the items in the list. The word “exemplary” is used exclusively herein to mean “serving as an example, instance, or illustration.” Any implementation described herein as “exemplary” is not necessarily to be construed as preferred or advantageous over other implementations.

[0250] The disclosure is not intended to be limited to the implementations shown herein. Various modifications to the implementations described in this disclosure may be readily apparent to those skilled in the art, and the generic principles defined herein may be applied to other implementations withoutdeparting from the spirit or scope of this disclosure. The teachings of the invention provided herein can be applied to other methods and systems, and are not limited to the methods and systems described above, and elements and acts of the various embodiments described above can be combined to provide further embodiments. Accordingly, the novel methods and systems described herein may be embodied in a variety of other forms; furthermore, various omissions, substitutions and changes in the form of the methods and systems described herein may be made without departing from the spirit of the disclosure. The accompanying claims and their equivalents are intended to cover such forms or modifications as would fall within the scope and spirit of the disclosure.

Claims

WHAT IS CLAIMED IS:

1. A scheduler for a communications network that transmits data over a plurality of heterogeneous underlay networks, the scheduler comprising:a network interface for communicating with the plurality of heterogeneous underlay networks;a non-transitory computer-readable medium storing processorexecutable instructions; anda processor communicatively coupled to the network interface and the non-transitory computer-readable medium, the processor-executable instructions configured to cause the processor to:rank the plurality of heterogeneous underlay networks from a highest priority to a lowest priority, the highest priority assigned to an underlay network of the plurality of heterogeneous underlay networks with a lower latency than the other underlay networks of the plurality of heterogeneous underlay networks;analyze a packet for transmission over an underlay network of the at least one of the plurality of heterogeneous underlay networks to determine a flow to which the packet belongs;responsive to determining that the packet belongs to a new flow, send the packet to the underlay network assigned the highest priority;determining a behavior of the flow to which the packet belongs;responsive to the behavior corresponding to a continuous throughput less than a throughput threshold rate, send the packet to the underlay network assigned the highest priority;responsive to the behavior corresponding to a burst size less than a burst threshold amount, send the packet to the underlay network assigned the highest priority; andresponsive to the behavior corresponding to a burst size greater than a burst threshold amount, send the packet to an underlay network assigned a lower priority lower than the highest priority.

2. The scheduler of claim 1, wherein the plurality of heterogeneous underlay networks includes a low earth orbit (LEO) satellite network.

3. The scheduler of claim 2, wherein the LEO satellite network corresponds to the underlay network with the lower latency than the other underlay networks of the plurality of heterogeneous underlay networks.

4. The scheduler of claim 3, wherein the plurality of heterogeneous underlay networks includes a geosynchronous orbit (GEO) satellite network.

5. The scheduler of claim 4, wherein the GEO satellite network has a higher latency than the LEO satellite network.

6. The scheduler of any one of claims 1 -5, wherein the lowest priority is assigned to an underlay network of the plurality of heterogeneous underlay networks with a longer latency than the other underlay networks of the plurality of heterogeneous underlay networks.

7. The scheduler of any one of claims 1-6, wherein determining the packet belongs to a new flow includes determining that the packet belongs to an idle-timed-out connection flow.

8. The scheduler of any one of claims 1-7, wherein determining the behavior of the flow to which the packet belongs includes employing a set of token bucket filters (TBFs) to assign a link priority to each packet of the flow, the link priority ranging from a highest link priority to a lowest link priority, the set of TBFs including a plurality of TBFs with a first TBF corresponding to the highest link priority and a second TBF corresponding to a link priority lower than the link priority of the first TBF.

9. The scheduler of claim 8, wherein the behavior corresponds to the continuous throughput being less than the throughput threshold rate when the set of TBFs assigns the packet the link priority corresponding to the highest link priority.

10. The scheduler of any one of claims 8 or 9, wherein the behavior corresponds to the burst size being less than the burst threshold amount when theset of TBFs assigns the packet the link priority corresponding to the highest link priority.

11. The scheduler of any one of claims 8-10, wherein the behavior corresponds to the burst size being greater than the burst threshold amount when the set of TBFs assigns the packet the link priority corresponding to a lower link priority than the highest link priority.

12. The scheduler of any one of claims 8-11 , wherein a number of token bucket filters in the set of TBFs is one less than a number of underlay networks of the plurality of heterogeneous underlay networks.

13. The scheduler of any one of claims 8-12, wherein a separate set of TBFs is used for each unique flow.

14. The scheduler of any one of claims 8-13, wherein a token fill rate of a first TBF of the set of TBFs corresponds to a stable throughput rate on the underlay network assigned the highest priority.

15. The scheduler of any one of claims 8-14, wherein an initial token size of the set of TBFs corresponds to a size sufficient to transmit initial burst data for the new flow to perform a slow start process.

16. The scheduler of any one of claims 8-15, wherein the set of TBFs is implemented by doing the following:set the first TBF as a current TBF to initiate a loop through the plurality of TBFs in the set of TBFs;responsive to determining that the current TBF has enough tokens to send the packet, assign to the packet the link priority corresponding to the current TBF; andresponsive to determining that the current TBF does not have enough tokens to send the packet, set a next TBF as the current TBF to continue the loop, the next TBF corresponding to a TBF with a next highest link priority.

17. The scheduler of any one of claims 1-16, wherein the processorexecutable instructions are further configured to cause the processor to responsive to determining that the underlay network assigned the highest priority is congested, send the packet to the underlay network assigned a next highest priority for packets determined to be sent to the underlay network assigned the highest priority.

18. The scheduler of any one of claims 1-17, wherein the processorexecutable instructions are further configured to cause the processor to determine flow statistic information for the flow to which the packet belongs.

19. The scheduler of any one of claims 1-18, wherein the processorexecutable instructions are further configured to cause the processor to implement a delay sensitivity classifier (DSC), the DSC configured to receive the packet and to set a delay-sensitivity level based on an analysis of the packet.

20. The scheduler of claim 19, wherein the delay-sensitivity level corresponds to an underlay network of the plurality of heterogeneous underlay networks.

21. The scheduler of any one of claims 19 or 20, wherein the DSC is further configured to set the delay-sensitivity level using a committed bucket refilled using a committed information rate and a peak bucket refilled using a peak information rate.

22. The scheduler of claim 21 , wherein the DSC is further configured to set the delay-sensitivity level by determining whether a length of the packet exceeds a size of the committed bucket and the peak bucket.

23. The scheduler of claim 22, wherein the DSC is further configured to set the delay-sensitivity level to a highest level responsive to determining that the length of the packet is not greater than the size of the committed bucket.

24. The scheduler of claim 23, wherein the DSC is further configured to set the delay-sensitivity level to a lower level than the highest level responsive to determining that the length of the packet is not greater than the size of the peak bucket.

25. The scheduler of claim 23, wherein the DSC is further configured to set the delay-sensitivity level to a lowest level responsive to determining that the length of the packet is greater than the size of the peak bucket.

26. The scheduler of claim 19, wherein the packet is sent to an underlay network based at least in part on the delay-sensitivity level set by the DSC.

27. A scheduler for a communications network that transmits overlay traffic over a plurality of heterogeneous underlay bearer networks having different latencies, the scheduler comprising:a network interface configured to communicate with the plurality of heterogeneous underlay bearer networks;a non-transitory computer-readable medium storing processorexecutable instructions; anda processor communicatively coupled to the network interface and the non-transitory computer-readable medium, the processor-executable instructions configured to cause the processor to:receive channel data frames corresponding to overlay packets associated with one or more flows;classify the channel data frames into delay-sensitive frames and throughput-sensitive frames based at least in part on behavioral characteristics of the flows;execute a delay sensitivity round in which delay-sensitive frames are preferentially scheduled for transmission over at least one underlay bearer network having a lower latency than at least one other underlay bearer network of the plurality of heterogeneous underlay bearer networks;execute a throughput sensitivity round subsequent to the delay sensitivity round in which throughput-sensitive frames are scheduled for transmission over at least one underlay bearer network having a higher latency than the at least one lower-latency underlay bearer network; andexecute a max bonded round subsequent to the throughput sensitivity round in which remaining throughput-sensitive frames arescheduled for transmission over any uncongested underlay bearer network of the plurality of heterogeneous underlay bearer networks to increase aggregate bonded capacity utilization,wherein scheduling decisions in each round are performed based at least in part on congestion control state information associated with each underlay bearer network.

28. The scheduler of claim 27, wherein the overlay traffic is transported between overlay endpoints using multi-path QUIC (MP-QUIC), and wherein the channel data frames correspond to QUIC stream frames encapsulated within bearer-specific QUIC connections established over the plurality of heterogeneous underlay bearer networks.

29. The scheduler of claim 28, wherein each underlay bearer network corresponds to a distinct QUIC path of the MP-QUIC connection, and wherein congestion control state information includes a congestion window and pacing rate maintained independently for each QUIC path.

30. The scheduler of any one of claims 27-29, wherein during the delay sensitivity round, transport acknowledgment packets are preferentially scheduled over the at least one lower-latency underlay bearer network to reduce effective control loop delay of an overlay transport protocol.

31. The scheduler of claim 30, wherein the transport acknowledgment packets include acknowledgment frames associated with a reliable overlay channel transported between overlay endpoints.

32. The scheduler of any one of claims 27-31, wherein the congestion control state information for each underlay bearer network includes at least one of: a transmission window size, a pacing rate, a measured round-trip time, a packet loss rate, or available transmission credits, and wherein scheduling in each round is conditioned on available transmission capacity indicated by the congestion control state information.

33. The scheduler of any one of claims 27-32, wherein the scheduler limits distribution of channel data frames belonging to a same overlay flow acrossunderlay bearer networks having substantially different round-trip times to reduce inter-packet delay variation.

34. The scheduler of claim 33, wherein the scheduler forwards channel data frames of a same overlay UDP flow to a same underlay bearer network except when required by congestion conditions identified during the max bonded round.

35. The scheduler of any one of claims 27-34, wherein within at least one of the delay sensitivity round, the throughput sensitivity round, or the max bonded round, channel data frames are selected for transmission according to a weighted fairness policy applied among multiple overlay flows.

36. The scheduler of claim 35, wherein the weighted fairness policy is based at least in part on a channel priority value or a configured traffic weight associated with the overlay flow.

37. The scheduler of any one of claims 27-36, wherein the plurality of heterogeneous underlay bearer networks includes:a low earth orbit (LEO) satellite network having a first latency; and a geosynchronous earth orbit (GEO) satellite network having a second latency greater than the first latency and superior bandwidth economics than the LEO satellite network.

38. The scheduler of claim 37, wherein the delay sensitivity round schedules delay-sensitive frames to the LEO satellite network and the throughput sensitivity round schedules throughput-sensitive frames to the GEO satellite network.

39. The scheduler of any one of claims 37 or 38, wherein acknowledgment packets associated with a bearer operating over the GEO satellite network are transmitted over the LEO satellite network to reduce an effective feedback delay associated with congestion control of the GEO satellite network.

40. An overlay network endpoint for a software-defined wide area network (SD-WAN) comprising:a plurality of bearer interfaces respectively associated with a plurality of heterogeneous access networks;a plurality of overlay channel modules configured to generate channel data frames corresponding to overlay UDP or TCP sessions;a congestion control module configured to maintain independent congestion control state for each bearer interface; anda round-based delay-sensitive classification and forwarding (DSC-F) scheduler configured to:in a first scheduling round, map delay-sensitive channel data frames to a bearer interface associated with a lower-latency access network;in a second scheduling round, map throughput-sensitive channel data frames to a bearer interface associated with a higher- latency access network with superior bandwidth economics; and in a third scheduling round, map remaining channel data frames to any bearer interface having available transmission capacity based on congestion control state,wherein the scheduler reduces inter-packet delay variation by limiting distribution of channel data frames from a same overlay flow across bearer interfaces having substantially different round-trip times.

41. The overlay network endpoint of claim 40, wherein the overlay channel modules are configured to support multi-path QUIC (MP-QUIC) sessions between the overlay network endpoint and a corresponding remote endpoint, and wherein each bearer interface corresponds to a respective QUIC path of an MP-QUIC connection.

42. The overlay network endpoint of claim 41 , wherein the congestion control module maintains, for each QUIC path, a congestion window and a pacing rate that are independently updated based on path-specific feedback.

43. The overlay network endpoint of any of claims 40-42, wherein the delay-sensitive channel data frames include transport acknowledgment frames, and wherein the scheduler preferentially maps the transport acknowledgmentframes to the bearer interface associated with the lower-latency access network to reduce control loop delay.

44. The overlay network endpoint of claim 43, wherein the transport acknowledgment frames correspond to acknowledgment messages of a reliable overlay channel transporting data frames between overlay network endpoints.

45. The overlay network endpoint of any of claims 40-44, wherein the congestion control state for each bearer interface includes at least one of: a transmission window size, a pacing rate, a measured round-trip time, a packet loss rate, or available transmission credits, and wherein execution of the first, second, and third scheduling rounds is conditioned on available transmission capacity indicated by the congestion control state.

46. The overlay network endpoint of any of claims 40-45, wherein the scheduler is configured to forward channel data frames belonging to a same overlay UDP flow to a same bearer interface except when congestion state indicates insufficient transmission capacity.

47. The overlay network endpoint of any of claims 40-46, wherein within at least one of the first scheduling round, the second scheduling round, or the third scheduling round, channel data frames are selected for transmission according to a weighted fairness policy applied among multiple overlay flows.

48. The overlay network endpoint of claim 47, wherein the weighted fairness policy is based at least in part on a channel priority value or a configured traffic weight associated with each overlay flow.

49. The overlay network endpoint of any of claims 40-48, wherein the scheduler is configured to identify a burst of packets associated with an overlay flow and to map a first predetermined number of bytes or packets of the burst to the bearer interface associated with the lower-latency access network during the first scheduling round.

50. The overlay network endpoint of any of claims 40-49, wherein the plurality of heterogeneous access networks includes a low earth orbit (LEO)satellite network and a geosynchronous earth orbit (GEO) satellite network, the LEO satellite network having a lower round-trip time than the GEO satellite network.

51. The overlay network endpoint of claim 50, wherein the scheduler maps delay-sensitive channel data frames to the bearer interface associated with the LEO satellite network and maps throughput-sensitive channel data frames to the bearer interface associated with the GEO satellite network.

52. The overlay network endpoint of any of claims 50-51, wherein acknowledgment packets associated with a bearer operating over the GEO satellite network are transmitted over the LEO satellite network to reduce effective congestion feedback delay for the GEO satellite network.

53. A method of scheduling packets for transmission over a plurality of heterogeneous underlay networks in a communications network, the method comprising:ranking the plurality of heterogeneous underlay networks from a highest priority to a lowest priority, the highest priority assigned to an underlay network having a lower latency than other underlay networks of the plurality;analyzing a packet to determine a flow to which the packet belongs; responsive to determining that the packet belongs to a new flow, sending the packet to the underlay network assigned the highest priority;determining a behavior of the flow to which the packet belongs; responsive to the behavior corresponding to a continuous throughput less than a throughput threshold rate, sending the packet to the underlay network assigned the highest priority;responsive to the behavior corresponding to a burst size less than a burst threshold amount, sending the packet to the underlay network assigned the highest priority; andresponsive to the behavior corresponding to a burst size greater than a burst threshold amount, sending the packet to an underlay network assigned a lower priority than the highest priority.

54. The method of claim 53, wherein the plurality of heterogeneous underlay networks includes a low earth orbit (LEO) satellite network and a geosynchronous earth orbit (GEO) satellite network, the LEO satellite network having a lower latency than the GEO satellite network and the GEO satellite network having a more economical capacity than the LEO satellite network.

55. The method of any of claims 53-54, wherein determining the behavior of the flow includes employing a set of token bucket filters (TBFs) to assign a link priority to the packet, the set of TBFs including a first TBF corresponding to a highest link priority and at least one additional TBF corresponding to a lower link priority.

56. The method of claim 55, wherein a separate set of token bucket filters is maintained for each unique flow, and wherein a token fill rate of the first TBF corresponds to a stable throughput rate of the underlay network assigned the highest priority.

57. The method of any of claims 53-56, further comprising, responsive to determining that the underlay network assigned the highest priority is congested, sending the packet to an underlay network assigned a next highest priority.

58. The method of any of claims 53-57, further comprising implementing a delay sensitivity classifier (DSC) configured to assign a delay-sensitivity level to the packet using a committed bucket refilled at a committed information rate and a peak bucket refilled at a peak information rate, and sending the packet to one of the plurality of heterogeneous underlay networks based at least in part on the delay-sensitivity level.

59. A method of forwarding overlay traffic over a plurality of heterogeneous underlay bearer networks having different latencies, the method comprising:receiving channel data frames corresponding to overlay packets associated with one or more flows;classifying the channel data frames into delay-sensitive frames and throughput-sensitive frames based at least in part on behavioral characteristics of the flows;executing a delay sensitivity scheduling round in which delaysensitive frames are preferentially transmitted over at least one underlay bearer network having a lower latency than at least one other underlay bearer network of the plurality;executing a throughput sensitivity scheduling round subsequent to the delay sensitivity scheduling round in which throughput-sensitive frames are transmitted over at least one underlay bearer network having a higher latency than the at least one lower-latency underlay bearer network; and executing a max bonded scheduling round subsequent to the throughput sensitivity scheduling round in which remaining throughputsensitive frames are transmitted over any uncongested underlay bearer network of the plurality to increase aggregate bonded capacity utilization, wherein scheduling in each round is performed based at least in part on congestion control state information associated with each underlay bearer network.

60. The method of claim 59, wherein the overlay traffic is transported between overlay endpoints using multi-path QUIC (MP-QUIC), and wherein the channel data frames correspond to QUIC stream frames encapsulated within bearer-specific QUIC connections established over the plurality of heterogeneous underlay bearer networks.

61. The method of claim 60, wherein each underlay bearer network corresponds to a distinct QUIC path of the MP-QUIC connection, and wherein the congestion control state information includes a congestion window and a pacing rate maintained independently for each QUIC path.

62. The method of any of claims 59-61, wherein during the delay sensitivity scheduling round, transport acknowledgment packets are preferentially transmitted over the at least one lower-latency underlay bearer network to reduce effective control loop delay of an overlay transport protocol.

63. The method of claim 62, wherein the transport acknowledgment packets include acknowledgment frames associated with a reliable overlay channel transported between overlay endpoints.

64. The method of any of claims 59-63, wherein the congestion control state information for each underlay bearer network includes at least one of: a transmission window size, a pacing rate, a measured round-trip time, a packet loss rate, or available transmission credits, and wherein transmission in each scheduling round is conditioned on available transmission capacity indicated by the congestion control state information.

65. The method of any of claims 59-64, further comprising limiting distribution of channel data frames belonging to a same overlay flow across underlay bearer networks having substantially different round-trip times to reduce inter-packet delay variation.

66. The method of any of claims 59-65, further comprising selecting channel data frames for transmission within at least one of the delay sensitivity scheduling round, the throughput sensitivity scheduling round, or the max bonded scheduling round according to a weighted fairness policy applied among multiple overlay flows.

67. The method of any of claims 59-66, wherein the plurality of heterogeneous underlay bearer networks includes a low earth orbit (LEO) satellite network having a first latency and a geosynchronous earth orbit (GEO) satellite network having a second latency greater than the first latency and superior bandwidth economics than the LEO satellite network.

68. The method of claim 67, wherein acknowledgment packets associated with a bearer operating over the GEO satellite network are transmitted over the LEO satellite network to reduce effective congestion feedback delay associated with the GEO satellite network.

69. A method of operating an overlay network endpoint of a software-defined wide area network (SD-WAN), the overlay network endpoint being communicatively coupled to a plurality of heterogeneous access networks through a plurality of bearer interfaces, the method comprising:maintaining independent congestion control state for each bearer interface;generating channel data frames corresponding to overlay UDP or TCP sessions;in a first scheduling round, mapping delay-sensitive channel data frames to a bearer interface associated with a lower-latency access network;in a second scheduling round, mapping throughput-sensitive channel data frames to a bearer interface associated with a higher-latency access network with superior bandwidth economics; andin a third scheduling round, mapping remaining channel data frames to any bearer interface having available transmission capacity based on the congestion control state,wherein the method reduces inter-packet delay variation by limiting distribution of channel data frames from a same overlay flow across bearer interfaces having substantially different round-trip times.

70. The method of claim 69, wherein the overlay UDP or TCP sessions are transported between the overlay network endpoint and a corresponding remote endpoint using multi-path QUIC (MP-QUIC), and wherein each bearer interface corresponds to a respective QUIC path of an MP-QUIC connection.

71. The method of claim 70, wherein maintaining independent congestion control state for each bearer interface includes maintaining, for each QUIC path, a congestion window and a pacing rate that are independently updated based on path-specific feedback.

72. The method of any of claims 69-71, wherein the delay-sensitive channel data frames include transport acknowledgment frames, and wherein mapping delay-sensitive channel data frames to the bearer interface associated with the lower-latency access network reduces control loop delay of an overlay transport protocol.

73. The method of claim 72, wherein the transport acknowledgment frames correspond to acknowledgment messages of a reliable overlay channel transporting data frames between overlay network endpoints.

74. The method of any of claims 69-73, wherein the congestion control state for each bearer interface includes at least one of: a transmission window size, a pacing rate, a measured round-trip time, a packet loss rate, or available transmission credits, and wherein execution of the first, second, and third scheduling rounds is conditioned on available transmission capacity indicated by the congestion control state.

75. The method of any of claims 69-74, further comprising forwarding channel data frames belonging to a same overlay UDP flow to a same bearer interface except when congestion state indicates insufficient transmission capacity.

76. The method of any of claims 69-75, further comprising selecting channel data frames for transmission within at least one of the first scheduling round, the second scheduling round, or the third scheduling round according to a weighted fairness policy applied among multiple overlay flows.

77. The method of any of claims 69-76, wherein the plurality of heterogeneous access networks includes a low earth orbit (LEO) satellite network and a geosynchronous earth orbit (GEO) satellite network, the LEO satellite network having a lower round-trip time than the GEO satellite network.

78. The method of claim 77, further comprising transmitting acknowledgment packets associated with a bearer operating over the GEO satellite network over the LEO satellite network to reduce effective congestion feedback delay associated with the GEO satellite network.

Images