Stateless Load Balancers for Low-Latency Stateful Connections
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
The existing network architectures in cloud provider networks incur excessive latency due to the use of multiple network function execution intermediaries (NFEIs) for message transfers between client programs and network-accessible services, which is particularly problematic for latency-sensitive applications.
Innovation Solution
Implementing networking managers (NMs) at virtualization hosts to generate and apply rules that bypass or replace NFEIs, using flow state information to establish low-latency paths directly between client programs and service request handlers, even in the presence of compute instance migrations.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Adaptability or versatility
If multiple network function execution intermediaries (NFEIs) are used for message transfers, then network function pipeline deployment flexibility is improved, but latency increases
Solution Approach 1:
The patent segments the network function pipeline into two distinct parts: (1) the initial connection establishment phase that uses NFEIs including load balancers for flexible deployment, and (2) the subsequent data transfer phase that uses direct low-latency paths bypassing NFEIs. This segmentation allows the system to enjoy deployment flexibility during connection setup while achieving low latency during actual data transfer.
Solution Approach 2:
The patent performs preliminary actions by establishing connection state information at NFEIs during the initial connection phase. The load balancer selects target instances and stores this state information, enabling subsequent direct connections without requiring NFEI intervention. This preliminary action allows the system to maintain deployment flexibility while eliminating latency for ongoing transfers.
2Device complexity
If stateless load balancers are used, then device complexity is reduced, but stateful connectivity cannot be maintained
Solution Approach 1:
The patent extracts the state maintenance function from the load balancer itself and relocates it to the network manager. The load balancer remains stateless, performing only simple connection routing based on received state information. The network manager assumes responsibility for maintaining connection state information, including target instance selections. This extraction allows the load balancer to remain simple while enabling stateful connectivity through the network manager's state management capabilities.
Solution Approach 2:
The patent introduces the network manager as an intermediary between the load balancer and the actual connection management. The load balancer sends connection requests to the network manager, which maintains the state information and provides appropriate responses. This intermediary architecture allows the load balancer to remain stateless while the system as a whole maintains stateful connectivity through the network manager's state management functions.
3Speed
If direct low-latency paths are established, then message transfer speed is improved, but adaptability to network function changes is reduced
Solution Approach 1:
The patent implements a feedback mechanism where the network manager continuously monitors the validity of established direct paths and NFEI chain configurations. When network function changes occur (such as instance migrations or load balancer reconfigurations), the network manager detects these changes and updates the appropriate paths or rules. This feedback loop ensures that direct low-latency paths are maintained when valid, while automatically adapting when network function changes require NFEI involvement.
Solution Approach 2:
The patent creates a dynamic system where the message transfer path is not fixed but can switch between direct low-latency paths and NFEI-based paths based on current network conditions and function validity. The network manager dynamically determines whether to use direct paths or NFEI chains, allowing the system to optimize for speed when possible while maintaining adaptability when network functions change.
Data Source
AI summary
A network connection is established between a client program and a particular request handler of a network-accessible service. The request handler is selected by a load balancer using an algorithm whose input includes configuration information of the service. The load balancer does not store flow state information for messages transmitted via the network connection. A networking manager of a host at which the client program runs stores the flow state information, including identification information of the particular request handler. Using the identification information, the networking manager causes additional messages directed to the service from the client program to be received at the particular request handler without using the load balancer as an intermediary.