Ticketing System for Cloud Job Scheduling and Fault Tolerance
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Managing large-scale administrative and operational tasks, such as backups and restores, in cloud computing environments is challenging due to the complexity of managing thousands of computing resources efficiently.
Innovation Solution
A ticketing system architecture that includes a master API, a pluggable scheduler, and stateless workers, allowing for efficient job scheduling and execution, with features like idempotent ticket management, redundancy, and horizontal scaling to ensure resilient operation across fault domains.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Ease of operation
If a centralized system is used to manage large numbers of computing resources, then coordination and control are simplified, but the system becomes a single point of failure and cannot tolerate network partitions or node failures
Solution Approach 1:
The system divides the centralized coordinator function into multiple independent coordinator nodes that can operate autonomously. Each coordinator can manage a subset of workers, and if one coordinator fails, others continue operating. This segmentation eliminates the single point of failure while maintaining coordinated control through distributed consensus mechanisms.
Solution Approach 2:
The patent introduces an intermediary layer between workers and coordinators through the use of tickets as standardized communication artifacts. Tickets serve as intermediaries that encapsulate job specifications, state information, and execution parameters, allowing workers to interact with any coordinator without requiring direct knowledge of specific coordinator identities or states.
2Reliability
If more coordinator nodes are added to improve reliability, then system availability increases, but the complexity of coordinating between multiple coordinators increases
Solution Approach 1:
All coordinator nodes implement identical multi-functional capabilities: they can create tickets, assign work to workers, update ticket states, and handle various event types. This universality means any coordinator can perform any function, simplifying the system architecture by eliminating specialized coordinator roles and reducing coordination complexity through symmetry.
Solution Approach 2:
The system changes the state parameters of tickets to track their progression through different phases (pending, in-progress, completed, failed). This parameter-based state machine approach provides a standardized interface for coordinating work across multiple coordinators, reducing complexity by enforcing consistent state transitions regardless of which coordinator is involved.
3Manufacturing precision
If work is assigned to specific workers to ensure proper execution, then job quality is maintained, but the system cannot scale horizontally and loses flexibility
Solution Approach 1:
The patent creates copies of work assignments through ticket replication. When a coordinator assigns work, it creates a ticket that can be independently processed by any worker. The ticket contains all necessary information to execute the job, allowing work to be copied and distributed across multiple workers without losing execution quality or flexibility.
Solution Approach 2:
Workers autonomously select and claim tickets from the coordinator without requiring explicit assignment. The worker evaluates its own capability to perform the work based on ticket requirements and self-selects appropriate tasks. This self-service mechanism maintains job quality through worker competency matching while enabling horizontal scaling as workers independently integrate into the system.
Data Source
AI summary
Examples include a ticketing system for managing performance of computer services. The ticketing system includes a ticket database to store a plurality of tickets, each ticket defining a job to be performed on a computing system, and a master to create a ticket, store the ticket in the ticket database, and update the ticket. The ticketing system includes a scheduler to determine whether to create the ticket based at least in part on a set of conditions, and to request the master to create the ticket when the ticket is to be created, and a worker to get the ticket from the master, perform the job defined by the ticket, and request the master to update the ticket with a status of the ticket. The ticketing system includes a supervisor to request the master to find a stale ticket, and, if found, to request the worker to abort performance of the job defined by the stale ticket.