Hybrid Pull-Push Job Scheduling for HPC and Big Data Workloads
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing large-scale computing systems face challenges in efficient resource management and job scheduling due to heterogeneity and scalability issues, leading to reduced resource utilization, throughput, and increased waiting times, particularly in systems with high-performance computing and big data processing jobs.
Innovation Solution
A combined Pull and Push mode resource management and job scheduling method that identifies job types and schedules high-performance computing jobs using a Push mode and big data processing jobs using a Pull mode, distributing jobs accordingly to optimize resource utilization and reduce master node workload.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Device complexity
If a centralized master node is used to manage all job scheduling in large-scale computing systems, then job scheduling control is centralized and simple to implement, but the master node becomes a performance bottleneck and single point of failure, reducing system scalability and resource utilization
Solution Approach 1:
The patent segments the centralized scheduling function into distributed scheduling nodes that operate autonomously. Each scheduling node manages a subset of computing resources and makes local scheduling decisions, eliminating the single master node bottleneck. This segmentation allows parallel processing of scheduling decisions across multiple nodes, significantly improving system throughput and scalability while maintaining scheduling control.
2Device complexity
If all jobs are scheduled using a single mode (either Push or Pull), then scheduling logic is simple and uniform, but resource utilization is suboptimal when dealing with heterogeneous job types including both high-performance computing and big data processing jobs
Solution Approach 1:
The patent implements dynamic scheduling mode selection that adapts to different job types. The system automatically switches between Push mode (for high-performance computing jobs requiring proactive resource allocation) and Pull mode (for big data processing jobs that can actively seek available resources). This dynamic approach optimizes resource utilization for heterogeneous workloads while keeping each individual scheduling logic relatively simple through mode-based abstraction.
Data Source
AI summary
Provided in the present invention are a Pull mode and Push mode combined resource management and job scheduling method and system, and a medium. The present invention includes parsing or identifying jobs to determine types, scheduling the jobs according to the types, and for high-performance computing jobs, distributing the jobs by using a Push mode: distributing computing nodes for the high-performance computing jobs, and pushing the high-performance computing jobs to the distributed computing nodes for execution; and for a big data processing job, using a Pull mode to perform distribution: waiting for a job request of each computing node, and when the computing node actively transmits the job request, transmitting the big data processing job to the computing node, which actively transmits the job request, for execution.