Adaptive Cloud Processing for Parallel Computing Workloads
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing parallel computing systems lack robust fault tolerance and adaptive scaling, particularly when using spot virtual machines, which can lead to job interruptions and inefficiencies due to worker or virtual machine failures and evictions.
Innovation Solution
A system that dynamically assigns and re-allocates tasks and virtual machines, detects failures, and replaces failed workers or virtual machines, ensuring continuous processing by utilizing spot virtual machines while maintaining auto-scaling and fault tolerance.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Loss of energy
If spot virtual machines are used for parallel computing jobs, then cost savings are achieved, but job interruptions and failures occur due to resource availability interruptions
Solution Approach 1:
The system pre-configures fault tolerance mechanisms including task re-submission policies, worker replacement strategies, and virtual machine reassignment rules before failures occur. When a spot VM is evicted or fails, the pre-established policies enable automatic recovery without manual intervention, maintaining job continuity while utilizing cost-effective spot instances
Solution Approach 2:
The system continuously monitors the status of workers and virtual machines, detecting failures and evictions in real-time. Based on this feedback, the system dynamically adjusts task assignments, triggers worker replacements, and reassigns tasks to available workers, ensuring continuous job processing despite the intermittent availability of spot VMs
2Productivity
If workers are assigned to tasks in parallel computing jobs, then processing efficiency is improved, but system complexity increases due to failure detection and replacement mechanisms
Solution Approach 1:
The system divides the parallel computing job into independent tasks that can be assigned to multiple workers. Each task is independently trackable and reassignable, allowing the system to segment the workload and manage failures at the task level rather than requiring system-wide complexity. This segmentation enables efficient parallel processing while keeping individual failure handling mechanisms simple
3Reliability
If failed workers are replaced and tasks are re-allocated, then job completion is maintained, but processing time increases due to failure recovery operations
Solution Approach 1:
The system maintains a pool of standby workers who are pre-positioned and ready to immediately take over failed workers' tasks. This preliminary preparation eliminates the need for time-consuming worker recruitment and task reassignment during failures, as standby workers can instantly assume failed workers' responsibilities and continue processing without interruption
Solution Approach 2:
When a worker fails, the system quickly discards the failed worker's task assignments and recovers by reassigning those tasks to available or standby workers. The task definitions and data are preserved, only the worker execution instance is replaced. This efficient recovery minimizes processing time loss while maintaining job completion integrity
Data Source
AI summary
The disclosed technology is generally directed to the processing of parallel computing jobs. In one example of the technology, for at least a first cluster of virtual machines that is assigned to a job, an initial assignment of at least one virtual machine to a parallel-computing job is made. Workers are assigned to tasks associated with the job. Upon failure of a task by one of assigned workers, the failed task is re-submitted. Upon detecting the failure of one of the workers assigned to the job, the failed worker is replaced with a replacement worker. Work associated with the failed worker is re-allocated to the replacement worker. Responsive to removal of a virtual machine assigned to the job, a new virtual machine is assigned to the job. Outputs are provided from the assigned workers.