Job Manager Platform for Cluster Computing Framework
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Current cluster computing frameworks face inefficiencies due to the constant operational need of the master device, which wastes resources and cannot manage interdependent jobs, retry failed jobs, or provide disaster recovery during cluster failures.
Innovation Solution
A job manager platform that remotely manages job execution by determining criteria, storing job information, and performing disaster recovery techniques, allowing for efficient execution and validation of jobs across a cluster computing framework.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If the master device operates constantly to manage job execution, then job scheduling and management functions are maintained, but computing resources are wasted
Solution Approach 1:
The master device transitions from constant operation to periodic operation, entering standby mode between job management tasks and only becoming active when jobs need scheduling or monitoring. This periodic activation pattern maintains job management reliability while dramatically reducing computing resource consumption during idle periods.
Solution Approach 2:
The job management functionality is extracted from the continuously running master device and consolidated into discrete task executions. The master device only performs management functions when specifically needed, rather than maintaining continuous operational readiness, thereby eliminating resource waste associated with constant operation.
2Ease of operation
If the master device manages all job execution tasks, then centralized control is maintained, but the system cannot manage interdependent jobs or provide disaster recovery
Solution Approach 1:
The job management system is segmented into independent worker devices that can autonomously manage their own execution and dependencies. Each worker device processes jobs independently, allowing the system to handle complex interdependent job scenarios while maintaining overall centralized coordination through the master device's task distribution capability.
Solution Approach 2:
Worker devices serve as intermediaries between the master device and the actual job execution processes. These intermediary worker devices provide the specialized functionality for managing interdependent jobs and disaster recovery, while the master device maintains centralized control through task allocation and monitoring.
3Reliability
If the master device handles all job execution and validation, then centralized monitoring is achieved, but execution speed and efficiency are reduced
Solution Approach 1:
The job execution and validation process is segmented across multiple worker devices, allowing parallel processing of multiple jobs simultaneously. Each worker device independently executes and validates jobs, eliminating the sequential bottleneck created by centralized master device processing while maintaining validation reliability through distributed verification.
Solution Approach 2:
Worker devices are empowered to autonomously execute and validate jobs without requiring continuous master device intervention. Each worker device performs self-validation of job completion and success, significantly improving execution speed while the master device maintains overall system coordination and receives final status reports.
Data Source
AI summary
A device, that provides serverless computing, receives a request to execute multiple jobs, and determines criteria for each of the plurality of jobs, wherein the criteria for each of the multiple jobs includes at least one of job posting criteria, job validation criteria, job retry criteria, or a disaster recovery criteria. The device stores information associated with the multiple jobs in a repository, wherein the information associated with the multiple jobs includes the criteria for each of the multiple jobs. The device provides a particular job, of the multiple jobs, to a cluster computing framework for execution, determines modified criteria for the particular job, and provides the modified criteria for the particular job to the cluster computing framework. The device receives, from the cluster computing framework, information indicating that execution of the particular job is complete, and validates a success of completion of the execution of the particular job.