Lock-Free Work-Stealing Scheduler for CPU Load Balancing
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Conventional thread scheduling mechanisms in multi-compute-engine systems fail to balance work across multiple CPUs, leading to CPU saturation and bottlenecks, resulting in lower I/O performance and performance variations between boot cycles or array nodes.
Innovation Solution
Implement a lock-free work-stealing thread scheduler that dynamically assigns threads to different CPUs using a shared ring buffer and a work-stealing algorithm, allowing idle CPUs to steal work from other CPUs and distribute polling tasks across all CPUs, eliminating the need for locks and reducing contention.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Productivity
If conventional thread scheduling mechanisms are used, then threads are assigned to CPUs, but work is not balanced across multiple CPUs leading to CPU saturation and bottlenecks
Solution Approach 1:
The patent segments the thread queue into per-CPU local queues and a shared global queue. Each CPU has its own local queue for threads it owns, while the shared queue stores threads that can be stolen by any idle CPU. This segmentation enables balanced work distribution by allowing idle CPUs to steal from the shared queue rather than waiting for a centralized scheduler.
Solution Approach 2:
The patent implements dynamic thread assignment where threads can move between CPUs through the work-stealing mechanism. Instead of static CPU-to-thread assignment, the system dynamically redistributes threads based on CPU availability and workload, allowing any idle CPU to steal threads from the shared queue and balance the workload in real-time.
2Reliability
If locks are used for thread scheduling, then thread assignment is controlled, but contention increases and performance decreases
Solution Approach 1:
The patent extracts the locking mechanism from the thread scheduling system. Instead of using locks to protect the thread queue and assignment logic, the system uses lock-free data structures and atomic operations to achieve thread assignment. This eliminates contention and performance penalties associated with lock acquisition and release.
Solution Approach 2:
The patent replaces the mechanical lock-based synchronization system with a lock-free work-stealing mechanism. Threads are assigned and stolen using atomic operations and memory ordering guarantees, eliminating the need for traditional locking hardware and software mechanisms, thereby reducing overhead and improving scalability.
3Reliability
If centralized thread scheduling is used, then thread assignment is managed, but scalability to large number of cores is limited
Solution Approach 1:
The patent segments the centralized scheduling authority into distributed per-CPU scheduling logic. Each CPU manages its own local queue independently, and the shared queue provides a common resource for thread allocation. This segmentation eliminates the bottleneck of a centralized scheduler and enables linear scalability with the number of cores.
Solution Approach 2:
The patent implements self-service scheduling where each CPU autonomously manages its own thread queue and actively participates in workload balancing by stealing threads from the shared queue when idle. This eliminates dependency on a centralized scheduler and enables each CPU to self-regulate based on local workload conditions, achieving excellent scalability.
Data Source
AI summary
Systems and methods are provided for lock-free thread scheduling. Threads may be placed in a ring buffer shared by all computer processing units (CPUs), e.g., in a node. A thread assigned to a CPU may be placed in the CPU's local run queue. However, when a CPU's local run queue is cleared, that CPU checks the shared ring buffer to determine if any threads are waiting to run on that CPU, and if so, the CPU pulls a batch of threads related to that ready-to-run thread to execute. If not, an idle CPU randomly selects another CPU to steak threads from, and the idle CPU attempts to dequeue a thread batch associated with the CPU from the shared ring buffer. Polling may be handled through the use of a shared poller array to dynamically distribute polling across multiple CPUs.