MapReduce Job Scheduling via Stage Duration Sorting
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Current MapReduce job scheduling policies, such as FIFO, do not always optimize resource utilization and completion time in batch workloads, leading to inefficient execution orders and increased processing time due to sequential execution of jobs.
Innovation Solution
A two-stage scheduler is introduced, which sorts jobs based on the minimum duration of their map or reduce stages and schedules them accordingly to minimize makespan, allowing for pipelined execution and efficient resource allocation by prioritizing jobs with shorter map stages first and longer reduce stages last.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Ease of operation
If FIFO scheduling policy is used, then simplicity of scheduling is maintained, but resource utilization efficiency deteriorates and completion time increases
Solution Approach 1:
The patent changes the scheduling parameter from simple FIFO ordering to a composite parameter that considers both map stage duration and reduce stage duration. This parameter change enables the scheduler to prioritize jobs strategically, improving resource utilization and reducing completion time while maintaining scheduling feasibility.
Solution Approach 2:
The patent introduces dynamic job ordering based on job characteristics (map and reduce stage durations) rather than static FIFO ordering. This dynamic approach adapts the execution order to the specific characteristics of each job, optimizing resource utilization and reducing overall completion time.
2Device complexity
If sequential execution of jobs is used, then resource allocation is simplified, but processing time increases due to lack of overlap
Solution Approach 1:
The patent performs preliminary analysis of job characteristics (map and reduce stage durations) before execution to determine the optimal execution order. This preliminary action enables strategic ordering that maximizes overlap between map and reduce stages, reducing processing time while maintaining manageable resource allocation complexity.
Solution Approach 2:
The patent ensures continuous utilization of computing resources by overlapping map stages of subsequent jobs with reduce stages of previous jobs. This continuous action eliminates idle time and maximizes resource utilization without significantly increasing allocation complexity.
3Ease of operation
If jobs are ordered without optimization, then scheduling is simpler, but makespan increases
Solution Approach 1:
The patent changes the scheduling criterion from simple arrival order to a composite parameter based on map and reduce stage durations. This parameter transformation enables the scheduler to minimize makespan by strategically ordering jobs, achieving up to 66% reduction in completion time while maintaining reasonable scheduling complexity.
Data Source
AI summary
Determining a schedule of a batch workload of MapReduce jobs is disclosed. A set of multi-stage jobs for processing in a MapReduce framework is received, for example, in a master node. Each multi-stage job includes a duration attribute, and each duration attribute includes a stage duration and a stage type. The MapReduce framework is separated into a plurality of resource pools. The multi-stage jobs are separated into a plurality of subgroups corresponding with the plurality of pools. Each subgroup is configured for concurrent processing in the MapReduce framework. The multi-stage jobs in each of the plurality of subgroups are placed in an order according to increasing stage duration. For each pool, the multi-stage jobs in increasing order of stage duration are sequentially assigned from either a front of the schedule or a tail of the schedule by stage type.