GPU Kernel Saving and Loading for Parallel Processing Efficiency
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Programming for graphical processing units (GPUs) is complex due to limited support for communications between threads and low memory bandwidth between GPU and host memory, making efficient data transfer and parallel algorithm execution challenging.
Innovation Solution
Systems and methods enable the saving and loading of GPU kernels and arrays to facilitate parallel processing across multiple GPUs, allowing for the combination of GPU kernels and arrays with parallel processing constructs like SPMD or PARFOR commands, enabling efficient execution on local or remote GPUs, including those in a cloud environment.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Ease of operation
If data is transferred frequently between GPU and host memory to enable efficient programming, then programming ease is improved, but memory bandwidth is exceeded and transfer time increases
Solution Approach 1:
The patent applies preliminary action by pre-transferring data to GPU memory before computation begins and pre-allocating memory resources. This allows the computational kernel to execute without waiting for data transfers, effectively hiding transfer latency behind computation time and eliminating the need for frequent synchronous transfers during execution.
Solution Approach 2:
The patent segments the computational task into distinct phases: data preparation on host, data transfer to device, kernel execution on GPU, and result retrieval. This segmentation allows independent optimization of each phase and enables overlapping of data transfer with computation in subsequent operations, reducing overall transfer time impact.
2Adaptability or versatility
If thread communication support is increased on GPU, then programming flexibility is improved, but device complexity increases
Solution Approach 1:
The patent uses shared memory as an intermediary mechanism for thread communication. Instead of implementing complex direct thread-to-thread communication hardware, threads communicate by reading and writing to shared memory variables, which provides sufficient flexibility for most parallel algorithms while keeping the hardware architecture relatively simple.
Solution Approach 2:
The patent employs copying mechanisms where data is copied between different memory spaces (host memory to device memory, global memory to shared memory) as needed. This copying approach provides flexibility in data movement and thread access patterns without requiring complex inter-thread communication infrastructure.
3Productivity
If memory bandwidth between GPU and host is increased, then data transfer efficiency is improved, but device cost increases
Solution Approach 1:
The patent maintains continuity of useful action by keeping data resident in GPU memory throughout the computational process. Once data is transferred to the device, it remains there for the duration of the kernel execution and subsequent operations, eliminating repeated transfers and maximizing the utilization of the available (though limited) memory bandwidth.
Solution Approach 2:
The patent performs preliminary data transfer and preprocessing operations before the main computational workload begins. This allows large datasets to be staged in GPU memory in advance, so that during the actual computation, the limited memory bandwidth is used for essential data access rather than repeated transfer operations.
Data Source
AI summary
A device receives, via a technical computing environment, a program that includes a parallel construct and a command to be executed by graphical processing units, and analyzes the program. The device also creates, based on the parallel construct and the analysis, one or more instances of the command to be executed in parallel by the graphical processing units, and transforms, via the technical computing environment, the one or more command instances into one or more command instances that are executable by the graphical processing units. The device further allocates the one or more transformed command instances to the graphical processing units for parallel execution, and receives, from the graphical processing units, one or more results associated with parallel execution of the one or more transformed command instances by the graphical processing units.