Cluster Scheduler Snapshot Updates Using Node Timing Deltas
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Cluster schedulers in large-scale clusters face significant calculation losses during snapshot updates and preemption processes, leading to crashes and inefficient scheduling algorithms.
Innovation Solution
Implement a method that uses a historical scheduling snapshot and a scheduling cache to record node timing values, selecting nodes with later timing values for incremental updates, optimizing snapshot updates and preemption processes through double-linked lists and balanced binary search trees.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If all node information is synchronized to scheduling snapshot during each scheduling, then scheduling completeness is improved, but calculation loss increases and scheduling stability deteriorates
Solution Approach 1:
The patent segments the node information synchronization process by dividing nodes into two categories: nodes with timing values later than the global timing value (需要同步的节点) and nodes with timing values not later than the global timing value (无需同步的节点). This segmentation allows the system to synchronize only the necessary portion of node information, reducing calculation loss while maintaining scheduling completeness for relevant nodes.
Solution Approach 2:
The patent applies partial action by performing synchronization only on nodes that have timing values later than the global timing value, rather than synchronizing all node information. This partial synchronization approach reduces the computational burden and calculation loss while still ensuring that the scheduling snapshot contains up-to-date information for nodes that have actually changed, thereby improving scheduling stability.
2Measurement precision
If complete snapshot synchronization is performed, then scheduling accuracy is improved, but scheduling throughput deteriorates
Solution Approach 1:
The patent segments the synchronization scope based on timing values, identifying and processing only those nodes whose timing values exceed the global timing value. This segmentation ensures that scheduling accuracy is maintained for changed nodes while avoiding unnecessary processing of unchanged nodes, thus preserving scheduling throughput.
Solution Approach 2:
The patent extracts and processes only the necessary node information (nodes with timing values later than the global timing value) from the complete node set. By taking out only the relevant changes and excluding unchanged nodes from the synchronization process, the system maintains scheduling accuracy for critical nodes while significantly improving scheduling throughput by reducing overall processing time.
3Productivity
If incremental update based on timing values is implemented, then update efficiency is improved, but system complexity increases
Solution Approach 1:
The patent implements preliminary action by pre-recording timing values for each node in the scheduling cache and pre-establishing the global timing value in the scheduling snapshot. This preliminary preparation enables the incremental update mechanism to efficiently identify which nodes need synchronization by simple comparison, thereby improving update efficiency while keeping the additional system complexity manageable through the use of straightforward timing value storage and comparison.
Data Source
Figure 1
Figure 2
Figure 3
AI summary
The present disclosure relates to a cluster-based scheduling method and apparatus, a medium, an electronic device, and a program product, which can reduce calculation loss, so that a cluster scheduler can ensure efficient and stable running in a large-scale cluster. The method includes: in response to a current scheduling, obtaining a historical scheduling snapshot generated in a previous scheduling and reading a global timing value corresponding to a cluster during the previous scheduling from the historical scheduling snapshot (S101), where the global timing value is a latest change timing of a node in the cluster during the previous scheduling recorded in chronological order; reading a respective timing value corresponding to each node in the cluster during the current scheduling from a scheduling cache and selecting, using the global timing value as a reference, a target node whose timing value is later than the global timing value in chronological order (S102); and updating the historical scheduling snapshot based on node information of the target node to obtain a scheduling snapshot of the current scheduling (S103).