Workload Pattern Scheduling Policy for Dynamic Batch Job Dependencies
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing job scheduler tools fail to dynamically identify job dependencies and optimal execution schedules for application batch workloads, requiring manual intervention and trial-and-error to achieve best performance results, especially in environments with varying hardware configurations and dynamic data volumes.
Innovation Solution
A method and system using reinforcement learning and fuzzy logic to generate a target pattern-based optimal scheduling policy, which automatically determines job execution times, machines, and dependencies based on historical telemetry data and metadata, reducing human intervention and optimizing job execution.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Productivity
If manual trial-and-error methods are used to define scheduling parameters, then performance optimization can be achieved, but time consumption and complexity increase significantly
Solution Approach 1:
The system enables self-service automation where the scheduling engine automatically defines and optimizes scheduling parameters (time slots, execution order, run machine allocation, dependencies) without requiring manual trial-and-error intervention. The engine uses historical telemetry data to autonomously generate optimal schedules, eliminating the time-consuming manual configuration process while maintaining high job execution efficiency.
Solution Approach 2:
The patent replaces the mechanical manual process of defining scheduling parameters with an automated computational system. The scheduling engine uses algorithms to automatically determine optimal schedules based on workload patterns and historical data, substituting human manual configuration with automated mechanical computation and decision-making processes.
2Adaptability or versatility
If static job definitions are used, then system simplicity is maintained, but adaptability to dynamic workload patterns is lost
Solution Approach 1:
The system transitions from static job definitions to dynamic scheduling by continuously analyzing historical telemetry data and workload patterns. The scheduling engine adapts its scheduling parameters based on observed patterns, allowing the system to dynamically respond to changing workload conditions while maintaining manageable complexity through automated pattern recognition and adaptation.
Solution Approach 2:
The system implements feedback mechanisms where historical telemetry data from job executions is continuously collected and analyzed. This feedback loop enables the scheduling engine to learn from past performance, identify workload patterns, and automatically adjust scheduling parameters to optimize future job execution, achieving adaptability without proportionally increasing system complexity.
3Extent of automation
If human intervention is used to monitor and finalize schedules, then control and accuracy are maintained, but automation level decreases
Solution Approach 1:
The patent replaces manual human monitoring and schedule finalization with automated computational processes. The scheduling engine automatically monitors performance metrics, analyzes telemetry data, and finalizes optimal schedules without human intervention, achieving high automation levels while maintaining measurement precision through systematic data collection and algorithmic analysis of job execution performance.
Data Source
AI summary
A method and a system for generating target pattern-based optimal scheduling policy for a set of jobs of an application is disclosed. The method includes: receiving, historical telemetry data associated with the set of jobs; generating, a workload pattern of the set of jobs based on the historical telemetry data; executing, a set of actions based on the workload pattern and a metadata store information of the set of jobs; generating, an intermediate policy based on the execution of the set of actions; executing, the set of jobs on a set of respective run machines based on the workload pattern and the intermediate policy; determining, a feedback data based on the execution of the set of jobs; and generating, the target pattern-based optimal scheduling policy based on at least one among the feedback data and the set of actions.


