Distributed Job Scheduling via Segmented Block Allocation
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
In distributed scheduling systems, as the number of triggers increases, existing methods often cause many triggers to miss their scheduled firing time, leading to missed job executions and other issues due to lock-based updating mechanisms.
Innovation Solution
Implementing a distributed scheduling approach where schedulers store and manage blocks of jobs in memory, using a distributed scheduling lock table to allocate and schedule jobs across multiple virtual machines, ensuring load balancing and failover capabilities.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Device complexity
If a lock-based updating mechanism is used for triggers in a distributed scheduling system, then the system can maintain simplicity in its scheduling approach, but as the number of triggers increases, many triggers miss their scheduled firing time resulting in missed job executions
Solution Approach 1:
The system segments the job table into multiple blocks and assigns each block to a specific scheduler instance. This segmentation eliminates the need for lock-based updating mechanisms by allowing each scheduler to independently manage its assigned block, thereby resolving the contradiction between simplicity and reliability when handling large numbers of triggers.
Solution Approach 2:
The system performs preliminary actions by pre-allocating blocks of jobs to scheduler instances before execution. This pre-allocation strategy ensures that schedulers have their assigned jobs ready in advance, preventing missed executions without requiring complex lock-based coordination during trigger firing.
2Device complexity
If multiple schedulers access and update the same job table concurrently using lock-based mechanisms, then the system can maintain a centralized scheduling approach, but this causes many triggers to miss their scheduled firing time
Solution Approach 1:
The job table is segmented into distinct blocks, with each block assigned to a specific scheduler instance. This segmentation transforms the centralized lock-based approach into a distributed block-based approach, where each scheduler independently manages its assigned block without requiring locks, thereby preventing triggers from missing their firing times.
Solution Approach 2:
The system introduces block assignments as an intermediary mechanism between the centralized job table and individual schedulers. This intermediary layer allows multiple schedulers to access and update the job table concurrently without conflicts, maintaining both the centralized structure and time accuracy.
3Device complexity
If a single scheduler manages all jobs in the job table, then the scheduling mechanism remains simple, but the system cannot handle high trigger loads efficiently without missing executions
Solution Approach 1:
The system segments the job table into multiple blocks and assigns each block to a different scheduler instance. This segmentation enables the system to handle high trigger loads efficiently by distributing the workload across multiple schedulers, thereby increasing job execution throughput while maintaining architectural simplicity through consistent block-based management.
Solution Approach 2:
The system merges multiple scheduler instances, each managing a block of jobs, into a coordinated distributed scheduling system. This merging approach increases overall productivity by enabling parallel processing of job blocks while maintaining the simplicity of individual scheduler operations through uniform block-based management.
Data Source
AI summary
A scheduling application is used to acquire into memory of a first processing system running on a computer system, a first jobs load for execution by the first processing system, the first jobs load stored in a database's table and having a first load size, the database associated with the computer system. The scheduling application is used to acquire into memory of a second processing system, a second jobs load for execution by a second processing system. The second jobs load being included in the table and having a second load size being balanced with the first load size and including jobs not in the first jobs load. In the first processing system's memory, jobs in the first jobs load are scheduled for execution by the first processing system. In the second processing system's memory, jobs in the second jobs load are scheduled for execution by the second processing system.