ML-Based Job Scheduling for Predictive Memory Allocation
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Conventional scheduling systems fail to predict fluctuations in memory requirements for application process jobs, leading to bottlenecks and inefficient use of resources due to insufficient temporary storage space.
Innovation Solution
A system that integrates a predictive engine-scheduling negotiator using machine learning algorithms to analyze historical job data, predict memory needs, and negotiate with service providers to allocate resources efficiently.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Productivity
If conventional scheduling systems allocate fixed memory resources, then system simplicity is maintained, but memory resource utilization efficiency deteriorates due to inability to predict fluctuations
Solution Approach 1:
The system performs preliminary actions by training machine learning models on historical job data before actual job scheduling. The trained models predict memory requirements in advance, allowing the scheduling system to allocate resources proactively rather than reactively, thereby improving utilization efficiency without requiring complex real-time adjustments
Solution Approach 2:
A machine learning prediction layer is introduced as an intermediary between historical data and the scheduling system. This intermediary processes historical job information and outputs predicted memory requirements, which then guide resource allocation decisions, effectively decoupling the complexity of prediction from the scheduling logic
2Productivity
If temporary storage space is increased to handle data processing, then processing capacity is improved, but resource waste increases when storage space is insufficient or over-provisioned
Solution Approach 1:
The system transitions from static fixed memory allocation to dynamic memory allocation based on predictions. Memory resources are adjusted dynamically according to predicted job requirements, allowing the system to optimize between having sufficient capacity and avoiding over-provisioning, thereby reducing resource waste while maintaining processing capacity
Solution Approach 2:
The scheduling system changes the parameter of memory allocation from a fixed value to a predicted variable based on historical patterns. By using machine learning models to determine optimal memory parameters for each job, the system adapts resource allocation to actual needs, improving processing capacity while minimizing waste
3Reliability
If conventional scheduling systems use fixed memory allocation, then system operation is simple, but service interruptions occur due to insufficient storage space
Solution Approach 1:
The system performs preliminary prediction of memory requirements using trained machine learning models before scheduling jobs. This advance prediction ensures that sufficient storage space is allocated beforehand, preventing service interruptions caused by insufficient memory, while the complexity is managed through automated model-based predictions rather than manual intervention
Data Source
AI summary
A system comprising databases, database processors, a job ingestion processor, and a processor. The databases store job information for previously performed jobs. Each previously performed job comprises timestamp, job name, and amount of memory used information. The database processors detect changes to job information of the previously performed jobs and, in response, communicate the job information associated with the affected jobs to a job ingestion processor. The job ingestion processor parses the job information into timestamp, job name, and amount of memory used variables for each affected job. The parsed job information comprises timestamp values, job name values, and amount of memory used values. The processor stores the job information variables in a data log. The processor uses the job information variables to train a machine learning algorithm and the trained machine learning model to predict amounts of temporary memory needed to execute job requests.