Resource Estimation Tool for Parallel Data Processing Optimization
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Current methods for predicting resource utilization in parallel data processing environments are inaccurate and require expertise, missing nuances in complex jobs and being time-prohibitive for optimization, especially for non-expert users.
Innovation Solution
A system and method that generates a resource utilization description using a resource estimation tool with modules for input data specification, data flow, resource utilization description, and performance reporting, allowing for alternative data flow graph scenarios and hardware recommendations.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Measurement precision
If current methods for predicting resource utilization are used, then expertise-based estimation is possible, but accuracy is poor and it is time-prohibitive for optimization
Solution Approach 1:
The patent creates a simplified copy or model of the parallel data processing job that can be analyzed without executing the full job. The resource utilization description module generates a model representation of the data flow graph and partition information, allowing prediction of resource needs through model analysis rather than actual job execution, thus saving time while maintaining accuracy
Solution Approach 2:
The system performs resource utilization prediction before the actual data processing job is executed. By analyzing the data flow graph and partition information in advance, the system provides resource estimates that allow for optimization of job parameters and hardware configuration prior to execution, eliminating the need for time-consuming trial executions
2Ease of operation
If current methods for predicting resource utilization are used, then estimation can be performed, but it requires expertise and nuances are missed
Solution Approach 1:
The resource estimation tool automatically analyzes the data flow graph and partition information without requiring user expertise. The system self-servingly extracts relevant parameters, applies the resource utilization model, and generates predictions, eliminating the need for expert intervention while maintaining high accuracy through systematic automated analysis
Solution Approach 2:
The patent introduces an intermediary resource estimation tool that acts as a bridge between the complex parallel data processing job and the user. This intermediary automatically handles the complex analysis of data flow graphs and partition information, translating technical complexities into usable resource predictions without requiring the user to understand the underlying nuances
3Productivity
If multiple scenarios are run to optimize job execution, then best order can be determined, but it is time prohibitive
Solution Approach 1:
The system performs preliminary analysis of the data flow graph to identify potential optimization scenarios without actually executing multiple job variants. By analyzing the model representation beforehand, the system can predict which execution orders or configurations will be most efficient, avoiding time-consuming trial executions
Solution Approach 2:
Instead of running multiple actual job scenarios to test different execution orders, the patent creates a model copy of the job and performs virtual scenario testing on this copy. The resource utilization description module can evaluate different data flow graph configurations and partitioning strategies in the model, providing optimization guidance without the time cost of actual scenario execution
Data Source
AI summary
An apparatus, system, and method are disclosed for generating a resource utilization description for a parallel data processing system. The method comprises developing a system model using the data flow graph for a target data processing job. The system model is then tuned based on a sample data set. The tuned system model is utilized to calculate the resource utilization description for the target data processing job. The resource utilization description may include disk space usage, scratch space usage, and processor utilization for each partition in the system. The method may further include determining a more efficient data flow graph to achieve the same results of the target data processing job.