Batch Job Fragmentation for Compute Instance Reliability
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Customers face challenges in executing large batch jobs quickly and reliably using compute instances due to resource constraints and the difficulty in managing applications across multiple virtual machine instances, which are not portable or scalable.
Innovation Solution
The approach involves subdividing batch jobs into job fragments that can be processed by separate compute instances, with control mechanisms to limit concurrent executions and manage resource usage, allowing for efficient processing and monitoring of job fragments within a time-bound compute service.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Productivity
If compute instances are used for batch jobs, then resource utilization is improved, but reliability deteriorates due to limited instance lifetime
Solution Approach 1:
The batch job is divided into multiple job fragments that can be independently executed by different compute instances. Each fragment is self-contained and can be processed separately, allowing the job to be distributed across multiple short-lived compute instances while maintaining overall job reliability through this modular approach.
2Productivity
If multiple virtual machine instances are used, then processing capacity is improved, but operational complexity worsens due to management requirements
Solution Approach 1:
Multiple job fragment tasks are merged into a single batch job execution workflow managed by the computing resource service. The service coordinates all virtual machine instances, handles application deployment uniformly, and provides centralized monitoring, thereby reducing operational complexity while maintaining high processing capacity through parallel execution.
3Speed
If batch jobs are subdivided into job fragments, then execution speed is improved, but system complexity increases due to control mechanisms
Solution Approach 1:
The batch job execution system implements self-service through automated orchestration. The computing resource service automatically divides the batch job into job fragments, assigns them to appropriate compute instances, monitors execution status, and handles failures without requiring complex manual control mechanisms. This automation maintains high execution speed while managing system complexity through intelligent self-coordination.
Data Source
AI summary
Systems and methods for the execution and operation of compute instances and tasks to perform batch jobs on behalf of customers of a computing resource service provider. Batch jobs can be subdivided into job fragments that can be concurrently executed by compute resources instantiated by the computing resource service provider. The computing resource service provider utilizes control mechanisms in the form of a maximum concurrent job fragment thresholds to mitigate the potential impact of resource constraints associated with larger batch jobs.