Distributed Job Execution Using Database Job Claiming
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Conventional distributed job execution systems without a central scheduler face challenges in ensuring non-overlapping data processing and preventing duplication, leading to wastage of computational resources.
Innovation Solution
A distributed job execution system that generates and stores job data records with status updates, allowing each container to act as a scheduler, ensuring non-overlapping processing by updating job status before execution and synchronizing with a database to avoid duplication.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If a central scheduler is used to manage distributed job execution, then job scheduling and status tracking are centralized and controlled, but the system becomes a single point of failure and consumes considerable resources
Solution Approach 1:
The patent extracts the scheduling functionality from a centralized scheduler and distributes it to individual containers. Each container maintains its own job data records and execution state, eliminating the single point of failure while preserving job management capabilities through local autonomy
Solution Approach 2:
The system segments the centralized scheduling function into distributed components where each container independently manages its own job execution. Job data records are partitioned across containers, with each container handling specific jobs without requiring a central coordinating authority
2Device complexity
If a central scheduler is removed from distributed job execution, then system complexity and resource consumption are reduced, but the system cannot ensure non-overlapping data processing and may duplicate job processing
Solution Approach 1:
The patent implements feedback mechanisms where containers continuously update job data records with their execution state (e.g., claiming jobs, processing status, completion). This feedback loop ensures that other containers can see the current state of jobs and avoid duplicating processing, maintaining precision without centralized control
Solution Approach 2:
Containers perform preliminary actions by claiming jobs and updating job data records before actual processing begins. This preliminary claim mechanism ensures that once a container takes a job, it is marked as processed, preventing other containers from duplicating the same job processing
3Ease of operation
If job status is not tracked before execution in a distributed system, then containers can access jobs freely, but duplicate processing occurs and computational resources are wasted
Solution Approach 1:
The patent requires containers to perform preliminary status updates and job claiming actions before executing job processing. By updating job data records to reflect claimed or in-progress status before actual processing, the system prevents duplicate execution while maintaining easy job access through the distributed database
Data Source
AI summary
Methods and systems are described herein for facilitating distributed job execution without a central job scheduler. The system may cause a container to, prior to executing job execution code for a job associated with a job data record, update a record instance of the job data record to indicate an updated status for the job and attempt to update the job data record at a database based on the record instance of the job data record. If the container successfully updates the job data record, the container may execute the job execution code for the job. If the container fails to update the job data record, the container may refrain from executing the job execution code for the job. The system may then update a first job data record associated with a first job at the database based on execution of the first job by a first container.