Aggregate Batch End Time Prediction Models
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Current batch processing systems face challenges in accurately predicting the end time of batch jobs, especially when dealing with large numbers of jobs that do not run in a static manner and require dynamic resource adjustments to meet target completion times.
Innovation Solution
A computer-implemented method that uses predictive analysis by aggregating newly generated batch end time predictive models after each milestone job finishes, forming an aggregate model to improve accuracy and stability of end time predictions, and dynamically updates predictions during job execution, allowing for what-if analysis to adjust resource usage.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Measurement precision
If traditional batch processing systems are used to predict end time, then the system structure is simple, but the prediction accuracy is insufficient
Solution Approach 1:
The batch job is divided into multiple milestone jobs, and separate predictive models are created for each milestone. This segmentation allows the system to track progress at different stages and improve overall prediction accuracy by aggregating multiple localized predictions rather than relying on a single complex model.
Solution Approach 2:
Multiple individual predictive models for different milestone jobs are merged into an aggregate predictive model. This combination leverages the strengths of each individual model to produce a more accurate overall end time prediction, resolving the contradiction by showing that combining simpler models can achieve better results than a single complex model.
2Productivity
If batch jobs run in static manner, then resource allocation is simple, but the ability to meet target completion times is reduced
Solution Approach 1:
The system transitions from static resource allocation to dynamic resource adjustment by continuously monitoring batch job progress through milestone completions and updating predictions accordingly. This allows the system to adapt resource allocation in real-time, improving completion efficiency while maintaining manageable complexity through structured adaptation at milestone points.
Solution Approach 2:
The system implements feedback mechanisms by monitoring actual milestone job completions and comparing them against predictions. This feedback loop enables dynamic adjustment of resource allocation and recalibration of predictive models, allowing the system to adapt to actual performance and improve productivity without requiring complete reconfiguration.
3Measurement precision
If predictive models are updated frequently, then prediction accuracy improves, but computational overhead increases
Solution Approach 1:
The predictive models are updated periodically at milestone job completions rather than continuously. This periodic update strategy maintains prediction accuracy by refreshing models at meaningful intervals while significantly reducing computational overhead compared to continuous updates, as the system only performs model aggregation when milestone events occur.
Data Source
AI summary
Performing predictive analysis on running batch jobs is provided. A series of batch end time predictive models is retrieved according to a sequence of milestone jobs in a batch of jobs. Retrieved batch end time predictive models are assembled into an aggregate batch end time predictive model to increase accuracy and stability of an end time prediction of the batch of jobs. The aggregate batch end time predictive model is utilized to predict an end time of the batch of jobs during running of the batch of jobs to form a predicted end time of the batch of jobs.