MapReduce Job Scheduling via Effective Processing Time
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
MapReduce clusters with heterogeneous computing nodes face inefficiencies due to varying computing capabilities and dynamic changes in available resources, leading to unfair and inefficient job processing under interfering workloads.
Innovation Solution
A scheduling system that calculates effective processing time for each computing resource by assigning weights based on processing speed and measuring available time, allocating resources to ensure jobs are completed at approximately the same time, and optimizing job placement across resources to maximize fairness and efficiency.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Productivity
If traditional scheduling mechanisms are used in MapReduce clusters with heterogeneous computing nodes, then jobs can be processed in parallel, but job completion times become unfair and inefficient due to varying computing capabilities and dynamic resource availability
Solution Approach 1:
The patent transforms the scheduling approach by changing from static resource allocation to dynamic allocation based on real-time parameters. It introduces effective processing time calculations that incorporate computing node capabilities, workload characteristics, and resource availability dynamics, allowing the scheduler to adapt parameters like task assignment and resource allocation based on current system state
Solution Approach 2:
The scheduling mechanism transitions from static to dynamic by continuously monitoring computing node availability, processing speeds, and workload conditions. The system adjusts task assignments in real-time based on changing resource conditions, ensuring that jobs are allocated to nodes that can complete them most efficiently at any given moment
2Ease of operation
If computing resources are allocated without considering heterogeneous capabilities, then resource allocation is simple, but fairness and efficiency of job processing deteriorate
Solution Approach 1:
The patent applies local quality by tailoring resource allocation strategies to the specific characteristics of each computing node. Instead of uniform treatment, the system evaluates individual node capabilities (processing speed, available memory, current workload) and assigns tasks accordingly, ensuring that each node contributes optimally based on its local properties
Solution Approach 2:
The scheduling system incorporates feedback mechanisms that continuously monitor job progress, node performance, and resource availability. This feedback loop allows the scheduler to learn from actual processing outcomes and adjust future allocations to improve fairness and efficiency while maintaining operational simplicity
3Device complexity
If static resource allocation is used, then scheduling is straightforward, but adaptability to dynamic workload conditions and resource availability is poor
Solution Approach 1:
The system performs preliminary actions by pre-calculating and pre-allocating resources based on predicted workload patterns and node capabilities. It prepares scheduling decisions in advance by evaluating multiple scenarios and pre-positioning tasks on nodes most likely to handle them effectively when conditions change
Data Source
AI summary
There is provided a method, a system and a computer program product for improving performance and fairness in sharing a cluster of dynamically available computing resources among multiple jobs. The system collects at least one parameter associated with availability of a plurality of computing resources. The system calculates, based on the collected parameter, an effective processing time each computing resource can provide to each job. The system allocates, based on the calculated effective processing time, the computing resources to the multiple jobs, whereby the multiple jobs are completed at a same time or an approximate time.