Batch Job Fragmentation for Transient Compute Instances
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Compute instances offered by service providers are challenging for customers to use for large batch jobs due to their transient nature, requiring complex management of multiple virtual machine instances and lack of portability and scalability.
Innovation Solution
A computing resource service divides batch jobs into job fragments that can be processed by time-bound compute instances, with each fragment having a pre-defined lifespan, allowing for efficient execution and monitoring to ensure completion or retry as needed.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Productivity
If compute instances are used for batch jobs, then resource cost is reduced and deployment speed is improved, but job reliability and completion guarantee deteriorate due to transient instance termination
Solution Approach 1:
The batch job is divided into multiple job fragments that can be independently executed. Each fragment is assigned to a compute instance with a specified lifespan, allowing the job to be processed in manageable chunks even as instances terminate and are replaced.
Solution Approach 2:
The system pre-configures compute instances with the necessary task definitions and job fragment assignments before execution begins. This preliminary setup ensures that when instances are launched, they are immediately ready to process their assigned fragments without delay.
2Productivity
If multiple virtual machine instances are used to process batch jobs, then processing capacity is increased, but system complexity and management overhead increase
Solution Approach 1:
A computing resource service acts as an intermediary between customers and the distributed compute instances. This service manages task definitions, monitors execution status, handles instance lifecycle events, and coordinates job fragment distribution, thereby abstracting away the complexity of managing multiple virtual machines from the customer.
Solution Approach 2:
The system implements automatic monitoring and retry mechanisms where compute instances self-report their execution status and the computing resource service automatically manages instance creation, termination, and job fragment reassignment without requiring customer intervention in the complex management tasks.
3Adaptability or versatility
If compute instances have limited lifespan, then resource flexibility and cost efficiency are improved, but job execution time and reliability deteriorate
Solution Approach 1:
The system ensures continuous job processing by automatically creating new compute instances to replace terminated ones and reassigning uncompleted job fragments. This maintains the useful action of job processing without interruption despite individual instance lifecycles, effectively extending the overall execution duration through continuous replacement rather than termination.
Data Source
AI summary
A computing resource service receives a request to execute a batch job. The computing resource service invokes a first compute instance to divide the batch job into a plurality of job fragments, where each job fragment comprises a respective subset of operations for the batch job that can be processed within a limited lifespan of a compute instance. The job fragments are distributed among a plurality of second compute instances to enable collective processing of the job fragments. The status of the batch job is determined based at least in part on individual statuses of the job fragments.