Computing Resource Recommendation Using Job Profiling and ML Estimators
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Managing computing resources in cloud environments is challenging due to the difficulty in estimating performance and cost-effectiveness of different system configurations, especially for customers who lack expertise and varied computing needs, and existing methods require hands-on expert analysis.
Innovation Solution
A system and method using machine learning to create a performance data matrix from historical job executions, enabling automatic recommendations for computing resources by profiling applications and systems, and updating the recommender system with new data over time.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Adaptability or versatility
If cloud computing vendors provide access to large numbers of systems with different configurations, then customers can select appropriate resources for their needs, but managing and recommending these resources becomes difficult due to the large number of variables
Solution Approach 1:
The patent creates a recommendation matrix that copies and stores performance data from previous job executions across multiple system configurations. This matrix serves as a reference model that captures the relationship between job characteristics and system performance, eliminating the need to analyze every possible configuration from scratch.
Solution Approach 2:
The system performs preliminary profiling by executing test jobs on various system configurations beforehand and storing the results in a recommendation matrix. This pre-computed data is then used to quickly recommend configurations for new jobs without requiring real-time analysis of all system variables.
2Measurement precision
If hands-on expert analysis is used for performance estimation, then accurate recommendations can be provided, but this requires significant expertise and is not easily available to cloud computing providers
Solution Approach 1:
The system enables self-service by automatically profiling jobs and generating performance estimates without requiring expert intervention. The recommendation matrix and automated algorithms allow the system to serve itself by making configuration recommendations based on stored historical data and job characteristics.
Solution Approach 2:
The patent replaces the mechanical process of expert analysis with an automated computational system. Instead of relying on human experts to manually analyze job performance, the system uses automated algorithms that process job characteristics and query the recommendation matrix to generate performance estimates.
3Adaptability or versatility
If customers select from multiple cloud computing vendors with different configurations, then more options are available, but customers need estimates of which configuration provides best performance, lowest price, or best value
Solution Approach 1:
The system performs preliminary performance evaluations by storing results from previous job executions in the recommendation matrix. When a customer submits a new job, the system quickly queries this pre-computed data to provide performance estimates, avoiding the need for time-consuming new evaluations.
Solution Approach 2:
The system incorporates feedback from actual job executions to continuously improve its recommendations. Performance data from real job runs is fed back into the recommendation matrix, allowing the system to learn from actual outcomes and refine its future recommendations for both performance and cost-effectiveness.
Data Source
AI summary
A system and method for recommending computing resources for processing jobs in a distributed computing environment with multiple heterogeneous computing resources are disclosed. Training applications or jobs are executed and measured on different computing resources and on different configurations of the computing resources to establish a database of performance metrics. A matrix of application features and computing resource features is created and populated with performance data. Machine learning may be used to create and update multiple recommendation engines based on the matrix that are cross-validated and merged to form final performance estimators. The performance estimators are applied to new applications and determine which existing applications are most similar and which resources to recommend.