Compute Kernel Scheduling via Data Transfer Pipelining
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Current scheduling approaches for compute kernels on heterogeneous accelerator devices fail to consider temporal and spatial relationships, leading to inefficiencies in data movement and compute invocation, resulting in suboptimal latency, throughput, and power performance due to inefficient data movement and memory thrashing.
Innovation Solution
The proposed solution involves cooperative full and partial data movement and kernel invocation pipelining, which optimizes data transfers and kernel invocations by temporally staggering data transfers across computing devices, avoiding concurrent data transfers and leveraging compilation and runtime systems for automatic workload optimization.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Ease of manufacture
If eager execution or lazy batching approaches are used for data movement and compute invocation, then implementation simplicity is maintained, but system-level optimization performance deteriorates due to lack of temporal and spatial relationship consideration
Solution Approach 1:
The system performs preliminary analysis of temporal and spatial relationships between compute tasks and data movement operations before execution. The compiler or runtime system pre-determines optimal scheduling by analyzing dependency graphs and resource availability, preparing optimized execution plans in advance that consider both computation and data transfer operations together, rather than handling them separately as in traditional eager or lazy approaches.
2Device complexity
If compute tasks scheduling and data movement are performed independently without considering temporal or spatial relationships, then scheduling complexity is reduced, but data movement efficiency deteriorates leading to memory thrash and data shuffle
Solution Approach 1:
The patent merges the scheduling of compute tasks with the scheduling of data movement operations into a unified optimization framework. The compiler or runtime system jointly analyzes compute dependency graphs and data flow requirements, creating an integrated schedule that coordinates when computations occur with when data is transferred, thereby eliminating independent scheduling overhead while preventing memory thrashing through coordinated resource management.
3Device complexity
If independent scheduling of compute tasks and data movement is used, then coordination overhead is minimized, but latency and throughput performance deteriorate due to inefficient resource utilization
Solution Approach 1:
The system achieves continuity of useful action by overlapping data movement operations with compute operations through intelligent scheduling. While data is being transferred between memory and processing units, compute operations that do not depend on that data are executed, maximizing resource utilization. The unified scheduling framework identifies opportunities for such overlap and coordinates them to maintain continuous productive work without increasing coordination complexity.
4Device complexity
If conventional scheduling methods are used without temporal and spatial relationship analysis, then power consumption management is simplified, but power efficiency deteriorates due to suboptimal performance
Solution Approach 1:
The system dynamically changes operational parameters such as clock frequencies, voltage levels, and processing modes based on the optimized schedule that considers temporal and spatial relationships. The unified scheduler identifies periods where data movement and computation can be overlapped or sequenced efficiently, and adjusts power management parameters accordingly to match actual workload demands, improving power efficiency without requiring complex dedicated power management mechanisms.
Data Source
AI summary
Examples relate to an apparatus, a device, a method, and a computer program for scheduling an execution of compute kernels on one or more computing devices, and to a computer system comprising such an apparatus or device. The apparatus comprises processing circuitry and interface circuitry. The processing circuitry is configured to determine an impending execution of two or more compute kernels to the one or more computing devices. The processing circuitry is configured to pipeline a data transfer related to the execution of the two or more compute kernels to the one or more computing devices via the interface circuitry.