Multicloud Sidecar Connectivity for NAT-Traversed Edge Clusters
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing methods for connecting containers across multiple Kubernetes edge clusters face challenges such as manual port configuration errors, complexity in modifying applications for ICE, and the need for public IP addresses, which are impractical for small edge clouds and IoT devices behind cascading NATs.
Innovation Solution
Utilizing ICE protocol with Init containers and sidecars to handle NAT traversal, avoiding triangular routing and application modifications, and leveraging existing STUN and TURN servers for direct end-to-end connectivity between edge clouds.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Ease of operation
If manual port configuration is used to connect containers across clusters, then connectivity can be established, but configuration errors and complexity increase
Solution Approach 1:
The system enables self-service automated connectivity configuration through the following mechanisms: (1) The Submariner component automatically discovers and configures VPN connections between clusters without manual intervention; (2) The system self-provisions network resources and establishes routing rules automatically; (3) Connectivity is established through automated peer-to-peer peer discovery and connection setup, eliminating manual port configuration errors while maintaining reliable connections
2Reliability
If ICE protocol is used for NAT traversal, then direct end-to-end connectivity can be achieved, but application modifications are required
Solution Approach 1:
The system introduces an intermediary approach by: (1) Deploying a VPN gateway component that handles NAT traversal protocols externally to applications; (2) The gateway acts as a mediator that establishes connections between clusters without requiring application-level ICE implementation; (3) The VPN infrastructure provides automated address discovery and connection establishment, achieving direct end-to-end connectivity while keeping applications unchanged
3Ease of operation
If public IP addresses are assigned to edge clouds, then direct access is enabled, but infrastructure cost and complexity increase
Solution Approach 1:
The system implements a virtualization-based approach where: (1) Virtual network interfaces and IP addresses are created and managed within the Kubernetes cluster through software-defined networking; (2) Virtual gateway components provide public-facing network endpoints without requiring physical public IP assignments to edge cloud infrastructure; (3) The virtual networking layer abstracts away the need for public IPs while enabling direct access capabilities through encrypted VPN tunnels
4Reliability
If VPN-based connectivity is established between clusters, then secure communication is achieved, but tunneling overhead increases
Solution Approach 1:
The system segments network traffic and VPN tunneling operations through: (1) Pod-specific network namespaces that isolate and optimize routing for each workload; (2) Selective encryption and tunneling only for inter-cluster traffic while leaving intra-cluster communication optimized; (3) Granular routing rules that direct traffic through VPN gateways only when necessary, reducing overall tunneling overhead while maintaining security for cross-cluster communications
Data Source
Figure 1~2
Figure 3
Figure 3
AI summary
A first edge node can communicate an address candidate for either a first sidecar container of the first edge node or a second sidecar container of a second edge node with a master orchestrator. The first edge node can then communicate traffic between a first application container of the first edge node and a second application container of the second edge node via a connection between the first sidecar container and the second sidecar container using the address candidate.