Distributed Job Claiming Without a Central Scheduler
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Conventional distributed job execution systems without a central scheduler face challenges in ensuring nonoverlapping data processing and preventing duplicate job processing, leading to wastage of computational resources.
Innovation Solution
A distributed job execution system that generates and stores job data records in a database, allowing each container to act as a scheduler, updating job statuses before execution, and synchronizing with the database to avoid duplicate processing, with mechanisms to handle failures and retries.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If a central scheduler is used to manage distributed job execution, then job scheduling and status tracking are centralized and controlled, but the system becomes a single point of failure and consumes considerable resources
Solution Approach 1:
The patent segments the centralized scheduler functionality into distributed components. Each container independently performs scheduler functions by attempting to claim jobs from the database and tracking their own execution status, eliminating the single point of failure while distributing the scheduling workload across multiple nodes
Solution Approach 2:
The patent extracts the scheduling logic from a separate central scheduler component and embeds it directly within each container. This allows containers to autonomously claim and execute jobs without relying on an external scheduling service, improving reliability while reducing overall system complexity
2Device complexity
If a central scheduler is removed from the distributed job execution system, then system complexity and resource consumption are reduced, but the system becomes unable to ensure nonoverlapping data processing and prevent duplicate job execution
Solution Approach 1:
The patent implements preliminary action by requiring containers to attempt to claim jobs from the database before executing them. The container updates the job status to 'in-progress' in advance, and other containers check this status before attempting to process the same job, preventing duplicate execution without requiring a central scheduler
Solution Approach 2:
The patent implements feedback mechanisms where containers continuously monitor the database for job status updates. When a container successfully claims a job, it updates the status in the database, and this feedback prevents other containers from processing the same job, ensuring accurate job processing distribution
3Productivity
If multiple containers process jobs simultaneously without coordination, then computational resources are utilized in parallel, but duplicate processing occurs and wastes computational resources
Solution Approach 1:
The patent introduces the database as an intermediary mechanism that coordinates job claims between multiple containers. Containers interact with the database to claim jobs and update status, which indirectly coordinates their processing activities and prevents duplicate execution while maintaining parallel processing capabilities
Solution Approach 2:
The patent implements self-service where each container independently manages its own job claiming and execution without requiring coordination from a central scheduler. Containers autonomously check the database for available jobs, claim them by updating status, and execute them, enabling parallel processing while preventing duplication through self-coordination
Data Source
AI summary
Methods and systems are described herein for facilitating distributed job execution without a central job scheduler. The system may cause a container to, prior to executing job execution code for a job associated with a job data record, update a record instance of the job data record to indicate an updated status for the job and attempt to update the job data record at a database based on the record instance of the job data record. If the container successfully updates the job data record, the container may execute the job execution code for the job. If the container fails to update the job data record, the container may refrain from executing the job execution code for the job. The system may then update a first job data record associated with a first job at the database based on execution of the first job by a first container.