ML Thread Estimation for Batch Job VM Selection
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
It is challenging for businesses to ensure sufficient computational resources are available for batch processing without overloading computer networks or wasting resources, as the exact resource requirements can vary based on the number of records being processed, making it difficult to reserve the right amount of resources in advance.
Innovation Solution
A machine learning-based approach is used to estimate the number of threads and select optimal cloud virtual machines for executing batch jobs within a maximum completion time, optimizing resource allocation by determining the most efficient resources needed to avoid excess resource usage.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If sufficient computational resources are reserved for batch processing, then the batch job can be completed within the required time period, but the computer network becomes overloaded and computational resources are wasted
Solution Approach 1:
The system dynamically determines the number of threads to execute based on the actual number of records in the batch job, rather than using a fixed reserved amount. This parameter adaptation allows the system to match computational resources to actual workload requirements, preventing both resource waste and network overload while ensuring timely completion.
Solution Approach 2:
The system uses machine learning models that are trained on historical batch job data to predict the number of threads needed. This feedback mechanism allows the system to learn from past performance and make informed decisions about resource allocation, improving both reliability and efficiency over time.
2Productivity
If the number of threads is increased to process more records faster, then the batch job completes within the time period, but the computational resources consumed increase
Solution Approach 1:
The system dynamically adjusts the number of threads as a parameter based on the actual batch job size and historical data patterns. This allows the system to optimize the balance between processing speed and resource consumption, using more threads only when necessary and fewer threads when the workload justifies it.
Solution Approach 2:
The thread allocation is made dynamic rather than static, allowing the system to adapt to varying workload conditions. The machine learning model continuously refines its predictions based on actual performance data, enabling the system to optimize resource utilization in real-time.
Data Source
AI summary
A method includes estimating a number of threads to execute batch job within maximum completion time by executing a thread estimation algorithm. Executing the thread estimation algorithm includes: determining, using an ML model, a collection of one or more combinations, each including an estimated number of threads and an actual completion time for processing the batch job using the estimated number of threads, the actual completion time being less than or equal to the maximum completion time, and saving the collection including the one or more combinations in a list of threads and completion times. The list of threads and completion times is used to select, based on a list of cloud virtual machines available for use, an optimal virtual machine or an optimal virtual machine combination, the optimal virtual machine or the optimal virtual machine combination being capable of executing the batch job within the maximum completion time.