Network Connector Scheduling for Large-Scale Node Reconnection
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing network connectivity solutions for large-scale computer deployments face performance and scalability issues during node reconnection, particularly in high availability and cluster system deployments, leading to communication disruptions and resource overruns.
Innovation Solution
A fast and scalable connector system utilizing a dual scheduler architecture with a master scheduler for global task management and a worker scheduler for concurrent execution, combined with a feedback learner for dynamic resource scaling and node-affinity based scheduling, to ensure efficient and resilient network reconnection.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Productivity
If multiple threads independently execute mass reconnect tasks in parallel, then reconnection performance is improved, but resource overruns and connection spikes occur
Solution Approach 1:
The reconnection task is divided into multiple priority levels (high, medium, low) and grouped by node affinity. The scheduler processes tasks in segments based on priority and affinity groups, preventing simultaneous connection spikes while maintaining parallel processing efficiency. This segmentation allows controlled parallel execution without resource overruns.
Solution Approach 2:
The scheduler dynamically adjusts thread pool sizes and connection rates based on real-time system state and node availability. Threads are created and destroyed dynamically rather than maintaining fixed pools, allowing the system to scale connection capacity up when needed and down to prevent resource overruns, thus balancing performance with reliability.
2Device complexity
If a single thread executes reconnection tasks sequentially, then implementation simplicity is maintained, but scalability and productivity are severely limited
Solution Approach 1:
The single-thread sequential execution is segmented into multiple priority levels and affinity groups. Instead of processing all nodes sequentially in one thread, the system segments tasks by priority and assigns them to multiple threads, maintaining simplicity in the high-level design while enabling parallel execution for improved throughput.
Solution Approach 2:
The system adds a priority dimension to the traditional single-thread sequential execution model. By introducing priority levels and affinity groups as additional sorting dimensions, the system transforms 1D sequential processing into a multi-dimensional scheduled execution model that enables parallelism without fundamentally complicating the implementation architecture.
3Productivity
If timing-wheel algorithm is used to interleave reconnecting nodes, then data structure efficiency is improved, but lack of advanced features and complete product solution
Solution Approach 1:
The timing-wheel algorithm is enhanced by segmenting reconnection tasks into multiple priority levels and affinity groups. This segmentation adds adaptability by allowing different timing strategies for different node types and scenarios, while maintaining the efficiency of the timing-wheel data structure for scheduling.
Solution Approach 2:
The system incorporates feedback mechanisms that monitor connection success rates, node availability, and system load. This feedback is used to dynamically adjust the timing-wheel parameters and scheduler behavior, adding adaptability and coordination capabilities while preserving the core timing-wheel efficiency for interleaved node reconnection.
Data Source
AI summary
A fast, scalable network connector that reliably and instantly creates link connections between management control plane and network device data plane nodes. A central database persists node states and provides a global connectivity view for system recovery. A master scheduler selects, prioritizes, and dispatches re-connect tasks. A multi-layer, elastic worker scheduler concurrently performs actual connection tasks through a socket I/O layer to the network nodes. The worker scheduler is scaled up or down as needed by the master scheduler. A feedback learner gathers information about node states and connectivity to provide insights that inform the scaling of the worker scheduler and scheduling of the re-connect tasks.