Consensus-Based Distributed Scheduling for Workload Continuity
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing distributed systems face challenges in managing workload performance due to unavailability of data processing systems, leading to potential suspension or termination of workload processing, especially in dynamic and challenging environments where connectivity changes frequently.
Innovation Solution
A consensus-based scheduling process is employed to select data processing systems for workload management, allowing for workload distribution that is independent of the availability status of individual systems, ensuring continuous workload processing even in environments with changing connectivity.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If a distributed system uses traditional workload scheduling methods, then the system structure is simple, but the workload processing is suspended or terminated when data processing systems become unavailable
Solution Approach 1:
The patent introduces a consensus-based scheduling mechanism as an intermediary layer between workload requests and data processing systems. This consensus mechanism coordinates scheduling decisions across multiple systems, ensuring that workload assignments are agreed upon by the majority, thereby preventing suspension when individual systems become unavailable.
Solution Approach 2:
The scheduling system dynamically adapts to the availability status of data processing systems by continuously monitoring system states and adjusting workload assignments in real-time. The consensus mechanism allows the system to flexibly reassign workloads when systems join or leave the distributed environment, maintaining operational continuity without rigid fixed assignments.
2Adaptability or versatility
If the system requires all data processing systems to be available for workload processing, then scheduling is simple, but the system cannot operate in dynamic environments with frequent connectivity changes
Solution Approach 1:
The consensus-based scheduling acts as an intermediary that abstracts the complexity of handling dynamic system availability. Instead of requiring direct coordination between all systems for every workload decision, the consensus mechanism mediates these interactions, allowing the system to adapt to environmental changes while maintaining a manageable level of operational complexity.
Solution Approach 2:
The system implements continuous feedback loops where data processing systems report their availability status to the consensus scheduling mechanism. This feedback enables the scheduler to dynamically adjust workload assignments based on current system states, allowing the system to adapt to frequent connectivity changes while maintaining coordinated scheduling decisions.
3Reliability
If a single data processing system is selected for workload management, then the scheduling process is simple, but the system creates a single point of failure
Solution Approach 1:
The patent segments the workload management function across multiple data processing systems rather than concentrating it in a single system. The consensus-based scheduling divides the decision-making authority among multiple nodes, so that no single point of failure can terminate workload processing. Each system contributes to the scheduling decision, and the workload can be reassigned if any individual system fails.
Data Source
AI summary
Methods and systems for managing workload performance in distributed systems is disclosed. The distributed system may include any number of data processing systems that may perform workloads. To manage workload performance, the distributed system may include a distributed control plane. The distributed control plane may include any number of data processing systems that both receive and service workload requests. When a workload request is received by one of the data processing systems of the control plane, a consensus based processing for a selecting one of the data processing systems to perform the workload may be performed. Consequently, the data processing system that received the workload request may or may not perform the workload to service the workload request depending on the outcome of the consensus based process.