Event-Driven DAG Job Scheduling for Distributed Systems
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing job scheduling systems in distributed computing environments face inefficiencies when managing large numbers of jobs with complex dependencies, as they often require repeated evaluation of entire graphs in response to events, leading to suboptimal execution schedules.
Innovation Solution
Implementing an event-driven scheduling system using directed acyclic graphs (DAGs) where only relevant portions of the graph are evaluated in response to specific events, such as job submission, execution, or completion, allowing for dynamic updates and efficient scheduling of jobs based on dependency relationships.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If repeated evaluation of entire graphs is performed in response to events, then scheduling completeness is maintained, but system efficiency and execution speed deteriorate
Solution Approach 1:
The patent segments the job scheduling system into two distinct components: a batch processing mode for initial comprehensive scheduling, and an event-driven mode for incremental updates. This segmentation allows the system to maintain scheduling completeness through the initial batch evaluation while improving efficiency by only re-evaluating affected portions of the graph in response to events, rather than repeatedly evaluating the entire graph.
Solution Approach 2:
The patent performs a preliminary comprehensive evaluation of the entire dependency graph in batch mode before transitioning to event-driven mode. This preliminary action establishes the initial scheduling state and identifies all jobs and their dependencies upfront, allowing subsequent event-driven updates to build upon this foundation without needing to re-evaluate the entire graph, thus maintaining reliability while improving productivity.
2Measurement precision
If comprehensive graph evaluation is performed, then all scheduling dependencies are accurately determined, but processing time and computational resources increase
Solution Approach 1:
The patent implements a dynamic scheduling approach that adapts the evaluation scope based on the current state and type of event. In batch mode, the entire graph is evaluated to ensure complete and accurate dependency determination. In event-driven mode, only the affected subgraph portions are re-evaluat ed, reducing processing time while maintaining accuracy for the changed elements. This dynamic adjustment of evaluation scope resolves the contradiction between comprehensive accuracy and processing efficiency.
3Productivity
If event-driven partial evaluation is implemented, then processing efficiency improves, but system complexity increases
Solution Approach 1:
The patent merges two scheduling approaches (batch processing and event-driven processing) into a unified system that leverages the strengths of both. The batch mode provides comprehensive initial scheduling, while the event-driven mode handles incremental updates efficiently. By combining these approaches and managing them through a consistent interface, the system achieves high processing efficiency without excessive complexity, as the two modes work complementarily rather than creating conflicting system components.
4Speed
If frequent graph re-evaluation is performed, then scheduling responsiveness to events improves, but resource consumption increases
Solution Approach 1:
The patent applies partial action by re-evaluating only the necessary portions of the dependency graph in response to events, rather than performing excessive full-graph evaluations. This approach maintains scheduling responsiveness by quickly identifying and processing affected jobs and dependencies, while conserving computational resources by avoiding redundant evaluations of unaffected portions of the graph.
Data Source
AI summary
Methods, systems, and computer-readable media for event-driven scheduling using directed acyclic graphs are disclosed. A directed acyclic graph is generated that comprises a plurality of nodes and a plurality of edges. The nodes represent jobs, and the edges represent dependency relationships between individual jobs. Based (at least in part) on one or more events, a job scheduler determines that one of the nodes represents a runnable job. One or more of the dependency relationships for the runnable job are satisfied by the one or more events. An execution schedule is determined for the runnable job. Based (at least in part) on the execution schedule, execution of the runnable job is initiated using one or more computing resources.