MapReduce Completion Time Estimation via Benchmark Workload Comparison
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Current methods lack a fast and efficient way to estimate completion times and costs for MapReduce jobs in cloud computing environments, where infrastructure configurations are hidden from users, leading to lengthy and complex processes.
Innovation Solution
A method is developed to build a general MapReduce performance model, compute performance characteristics of benchmark workloads, select workloads with similar characteristics, and estimate completion times by comparing these in known and unknown hardware configurations, using a comparative analysis.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Measurement precision
If the MapReduce job is run on each cluster to obtain completion time estimates, then the accuracy of the estimate is improved, but the time required to obtain the estimate increases considerably
Solution Approach 1:
The patent creates a virtual copy of the MapReduce job configuration and submits it to a test cluster to measure performance characteristics. This copy approach allows obtaining accurate completion time estimates without running the actual production job, thus resolving the contradiction between estimate accuracy and time consumption.
Solution Approach 2:
The patent performs preliminary performance measurements on test clusters using the same or similar configurations before actual job execution. By pre-measuring performance characteristics of virtual nodes and computing resources, the system can estimate completion times without waiting for actual job execution, thereby reducing the time to obtain estimates while maintaining accuracy.
2Adaptability or versatility
If the MapReduce job is run on multiple virtual nodes with different configurations to obtain comprehensive estimates, then the versatility and accuracy of estimates are improved, but the complexity and time required increase
Solution Approach 1:
The patent develops a universal performance model that can estimate completion times across multiple virtual node configurations without requiring separate measurements for each. The model uses performance characteristics from test clusters to generalize estimates for different virtual machine types (e.g., m1.small, m1.medium, m1.large), thus achieving versatility without proportional increases in complexity.
Solution Approach 2:
The patent measures performance characteristics by varying key parameters such as virtual machine configuration, data size, and job complexity. By establishing relationships between these parameters and completion times through controlled experiments, the system can predict performance across different configurations without running exhaustive tests on each combination, thereby managing complexity while maintaining adaptability.
Data Source
AI summary
A method, non-transitory computer readable medium, and apparatus for estimating a completion time for a MapReduce job are disclosed. For example, the method builds a general MapReduce performance model, computes one or more performance characteristics of each one of one or more benchmark workloads, computes one or more performance characteristics of the MapReduce job in the known processing system, selects a subset of the one or more benchmark workloads that have similar performance characteristics as the one or more performance characteristics of the MapReduce job, targets a cluster of processing nodes in a distributed processing system, computes one or more performance characteristics of the subset of the one or more benchmark workloads in the cluster of processing nodes and estimates the completion time for the MapReduce job.