Convergence Scheduling via Ticketing System for Cloud Resilience
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Managing large-scale administrative and operational tasks, such as backups and restores, in cloud computing environments is inefficient due to the complexity of scheduling and resilience in the face of failures and network partitions.
Innovation Solution
A ticketing system architecture with a master API, pluggable scheduler, supervisor, and stateless workers that use convergence-based scheduling and redundancy to ensure resilient and efficient job management, allowing for independent operation and fault tolerance.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If traditional scheduling systems are used to manage large-scale administrative and operational tasks, then the system can handle basic scheduling requirements, but the system becomes inefficient and unreliable in the face of failures and network partitions
Solution Approach 1:
The system is divided into independent components: supervisors, workers, and a ticketing system. Each supervisor manages its own set of workers and operates independently. This segmentation allows individual components to fail without affecting the entire system, thereby improving reliability while maintaining manageable complexity through modular design.
Solution Approach 2:
A ticketing system acts as an intermediary between supervisors and workers. Tickets encapsulate job information, state, and metadata, serving as a standardized communication interface. This intermediary layer decouples the supervisor-worker interaction, allowing independent operation and improving system reliability during failures and network partitions.
2Ease of operation
If centralized scheduling control is used to coordinate tasks, then task coordination is simplified, but the system becomes vulnerable to single points of failure and network partitions
Solution Approach 1:
Centralized scheduling control is segmented into distributed supervisors, each managing its own workers independently. This eliminates the single point of failure inherent in centralized systems while maintaining operational simplicity through standardized ticket-based communication. Each supervisor can operate autonomously, improving fault tolerance without sacrificing ease of operation.
Solution Approach 2:
Supervisors and workers are stateless and self-sufficient, obtaining job information from tickets rather than maintaining complex state information. This self-service approach allows components to operate independently without relying on continuous centralized coordination, thereby improving fault tolerance while keeping the system easy to operate through standardized interfaces.
3Device complexity
If stateful workers are used to maintain job state information, then job tracking is simplified, but the system becomes less resilient to worker failures and harder to scale
Solution Approach 1:
Instead of maintaining state in memory (stateful workers), the system uses tickets as persistent copies of job information stored in a durable ticket store. Each ticket contains all necessary job state information, allowing workers to be stateless and easily replaceable. If a worker fails, another worker can pick up the ticket and continue execution, improving resilience without increasing operational complexity.
Solution Approach 2:
Workers are designed to be stateless and transient rather than stateful and persistent. This allows workers to be easily created, destroyed, and replaced without losing job state information, which is preserved in the ticket store. This approach improves resilience to worker failures and makes the system more scalable, while the ticketing system maintains straightforward job tracking.
4Ease of manufacture
If time-based scheduling is used to dispatch tasks, then scheduling is simple to implement, but the system becomes unreliable when workers are unavailable or fail
Solution Approach 1:
The system implements feedback through the ticket state machine and supervisor-worker interaction. Supervisors continuously monitor ticket states and worker availability, adjusting scheduling decisions based on current system conditions rather than relying solely on pre-determined time schedules. This feedback mechanism improves reliability by adapting to worker availability while maintaining relatively simple scheduling logic through standardized ticket processing.
Solution Approach 2:
The scheduling system transitions from static time-based scheduling to dynamic event-driven scheduling. Tickets move through a state machine based on events such as worker availability, job completion, and failures. This dynamic approach improves reliability by responding to actual system conditions while maintaining ease of implementation through the structured ticket state machine and standardized supervisor-worker protocols.
Data Source
AI summary
Examples include a computer-implemented method for determining one or more observations about a computing system, determining a plurality of sets of one or more actions based at least in part on a plurality of policies and the one or more observations; reducing the plurality of sets of one or more actions to generate a reduced set of actions, the reduced set of actions including scheduling a computing system to execute the reduced set of actions by requesting one or more idempotent tickets to be created in a ticketing system; and performing, or causing to be performed, the reduced set of actions.