Hardware-Assisted Data Movement via Proxy Object Scheduling
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Memory management techniques in data processing systems that transfer data structures between GPUs and CPUs are inefficient due to high processing overhead caused by slower memory bandwidth between CPUs and GPUs, leading to significant time spent copying data.
Innovation Solution
The implementation of task-to-proxy-object scheduling, which schedules the transfer of proxy objects representing sub-data-blocks from CPU memory to GPU memory during the execution of predecessor tasks, overlapping memory transfers with task executions to reduce memory transfer time.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If entire data structures are copied from CPU memory to GPU memory outright, then data availability for GPU compute kernels is ensured, but memory transfer time increases significantly due to slower interconnect bandwidth
Solution Approach 1:
The patent segments data structures into smaller sub-data-blocks that can be transferred individually. Instead of copying entire data structures at once, the system divides them into manageable chunks that can be transferred in parallel during different task executions, reducing total transfer time while ensuring data availability.
Solution Approach 2:
The patent implements preliminary action by scheduling memory transfers of sub-data-blocks to occur during the execution of predecessor tasks. This allows data to be prepared and transferred in advance before the current task needs it, overlapping transfer operations with computation operations to eliminate idle waiting time.
2Productivity
If data is copied back from GPU memory to CPU memory after compute kernels, then processing results are retrieved, but processing overhead increases due to bandwidth limitations
Solution Approach 1:
The patent ensures continuity of useful action by scheduling memory transfer operations to occur during task executions rather than sequentially before or after. This overlapping allows memory transfers and compute operations to proceed simultaneously, keeping the system continuously productive without idle waiting periods.
Solution Approach 2:
The patent applies preliminary action to data transfer operations by initiating them during predecessor task executions. This allows results to be prepared for transfer in advance, and the transfer to occur during computation of subsequent tasks, eliminating the sequential bottleneck and reducing overall processing overhead.
3Speed
If memory bandwidth between CPU and GPU is increased, then data transfer speed improves, but system complexity and cost increase
Solution Approach 1:
The patent segments data transfers into multiple smaller operations that can be performed in parallel during different time periods. This approach achieves high effective transfer throughput without requiring a single high-bandwidth interconnect, as the cumulative effect of multiple parallel transfers matches the performance of a faster but more complex single connection.
Solution Approach 2:
The patent uses preliminary action to schedule transfers during computation execution, effectively utilizing available bandwidth without requiring higher physical bandwidth. By preparing data transfers in advance and overlapping them with computation, the system achieves high data movement efficiency through scheduling rather than through increased hardware complexity.
Data Source
AI summary
A processor includes a task scheduling unit and a compute unit coupled to the task scheduling unit. The task scheduling unit performs a task dependency assessment of a task dependency graph and task data requirements that correspond to each task of the plurality of tasks. Based on the task dependency assessment, the task scheduling unit schedules a first task of the plurality of tasks and a second proxy object of a plurality of proxy objects specified by the task data requirements such that a memory transfer of the second proxy object of the plurality of proxy objects occurs while the first task is being executed.