Locality-Aware Resource Allocation for Map-Reduce Jobs
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Map-reduce jobs in cloud computing generate significant network traffic due to inefficient data transfer between nodes during the shuffle-phase, leading to increased network load and reduced performance.
Innovation Solution
Implementing a locality-aware resource allocation system that determines the volume of input data for the map-phase and reduce-phase of a map-reduce job, selectively placing data blocks and virtual machines to minimize network hops, thereby optimizing data transfer and reducing network traffic.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Device complexity
If data blocks and virtual machines are placed without locality awareness, then resource allocation is simpler, but network traffic increases and job execution time increases
Solution Approach 1:
The system performs preliminary placement of data blocks and virtual machines on computing nodes before map-reduce job execution begins. By determining the volume of input data and pre-positioning resources according to locality requirements, the system avoids complex runtime resource allocation while minimizing network hops during actual job execution, thus reducing overall job execution time without adding operational complexity
2Loss of energy
If data blocks and virtual machines are selectively placed for locality awareness, then network traffic decreases, but resource allocation complexity increases
Solution Approach 1:
The system applies local quality by placing data blocks and virtual machines together on the same computing nodes based on locality requirements. This localized placement strategy ensures that map-phase and reduce-phase operations access data with minimal network hops, significantly reducing network traffic. The placement decision is made based on the specific volume of input data for each job phase, creating optimized local configurations without requiring system-wide complexity
Data Source
AI summary
Computing resource allocation for map-reduce job execution comprises determining the volume of input data to the map-phase and the reduce-phase of a map-reduce job prior to execution. Based on said determination, data blocks and virtual machines (VMs) are selectively placed for locality aware map-reduce job execution on a cluster of computing nodes in a network. Selectively placing data blocks and VMs comprises integrally placing the data and the VMs at selected nodes to lower data transfer network hops for a map-phase and a shuffle-phase of the map-reduce job upon execution by the VMs.