GPU Memory Expansion via NVMe Data Migration Paths
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Current solutions for expanding GPU memory, such as model parallelism, data compression, unified memory architecture, and unified memory architecture with file I/O, are inadequate in GPU multi-tenant mode and when processing ultra-high-resolution images, as they introduce latency, are limited by host memory size, or require manual memory management.
Innovation Solution
A method that utilizes a unified memory architecture with an external storage device, like solid-state disks supporting NVMe, to migrate data from storage to GPU memory upon page faults, offering transparent expansion through candidate paths such as cache or direct data migration, optimizing for bandwidth and latency.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Quantity of substance
If model parallelism is used to expand GPU memory, then the memory capacity is increased, but latency is introduced and manual memory management is required
Solution Approach 1:
The system segments memory into multiple levels: GPU memory for frequently accessed data and external storage devices for less frequently accessed data. This segmentation allows the system to maintain high-speed access for critical data while expanding total capacity using external storage, resolving the contradiction between memory capacity and access latency.
Solution Approach 2:
The system introduces a memory management unit (MMU) and virtual memory management software as intermediaries between the GPU and external storage. These intermediaries automatically handle data migration between GPU memory and external storage based on access patterns, eliminating the need for manual memory management while expanding effective memory capacity without significant latency penalty.
2Ease of operation
If data is migrated from storage device to GPU memory via host cache, then data migration is achieved, but additional latency is introduced compared to direct migration
Solution Approach 1:
The system dynamically selects the data migration path based on real-time conditions. When the GPU needs data, the MMU determines whether to migrate through host cache or directly from storage device, adapting to current system state. This dynamic approach optimizes migration latency while maintaining ease of operation, as the selection is automatic and condition-based.
Solution Approach 2:
The system changes the migration path parameter dynamically based on access patterns and system state. By monitoring whether data is likely to be reused soon, the system adjusts the migration strategy - using direct migration for one-time access or cache-mediated migration for potentially reused data - thereby optimizing latency while preserving operational simplicity.
3Quantity of substance
If unified memory architecture with file I/O is used, then memory expansion is achieved, but manual memory management is required which reduces ease of operation
Solution Approach 1:
The system implements self-service memory management through the MMU and virtual memory management software that automatically monitor GPU memory usage patterns and initiate data migration between GPU memory and external storage without programmer intervention. This self-service mechanism maintains expanded memory capacity while eliminating manual management requirements, improving ease of operation.
Solution Approach 2:
The system uses feedback from GPU memory access patterns to automatically control data migration. The virtual memory management software continuously monitors which data is being accessed and adjusts migration decisions accordingly, creating a closed-loop system that automatically optimizes memory usage without requiring manual intervention from programmers while maintaining expanded capacity.
Data Source
AI summary
Techniques are disclosed for expanding a memory of a graphics processing unit. For example, a method comprises: receiving, from a graphics processing unit coupled to a host, an indication that data to be accessed by the graphics processing unit is absent in the memory of the graphics processing unit, where the data is stored in a storage device coupled to the host; in response to receiving the indication, determining, from a plurality of candidate paths, a path for migrating the data from the storage device to the memory of the graphics processing unit; causing the data to be migrated from the storage device to the memory of the graphics processing unit via the determined path; and instructing the graphics processing unit to access the data from the memory of the graphics processing unit. As such, a memory of a GPU can be expanded with an external storage device.