Unified Virtual Memory Weight Data Pinning for GPU Cache Access
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing unified virtual memory (UVM) technologies face performance limitations due to their fault-based architecture, leading to significant overhead in data migration and communication between CPU and GPU, especially during GPU oversubscription events.
Innovation Solution
The method involves checking properties of data blocks in the UVM used for deep learning models, preferentially storing weight data in the main processor memory, and performing operations using the coprocessor while directly loading data from the main processor memory into the coprocessor cache, thereby preventing data migration and optimizing memory usage.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Adaptability or versatility
If data is migrated between CPU memory and GPU memory in existing UVM technologies, then data access flexibility is improved, but fault handling overhead increases significantly
Solution Approach 1:
The patent applies preliminary action by pinning weight data to CPU memory before GPU execution. This pre-positioning of data eliminates the need for fault handling during GPU operations, as the data remains in CPU memory and is directly accessible to the GPU without triggering page faults. The pinning operation performs the data placement in advance, preventing the harmful fault handling overhead that would otherwise occur during model execution.
Solution Approach 2:
The patent extracts the weight data from the unified virtual memory management system and places it in a dedicated pinned state in CPU memory. This separation removes weight data from the dynamic page migration mechanism, eliminating the source of page faults for this critical data type. By extracting weight data from the general UVM pool and giving it special status, the system eliminates fault handling overhead for weight access while maintaining flexibility for other data types.
2Productivity
If GPU memory capacity is increased to handle larger models, then model processing capability is improved, but memory cost and device complexity increase
Solution Approach 1:
The patent merges CPU memory and GPU memory into a unified virtual address space that can be accessed by both processors. This unified memory architecture allows the system to use CPU memory as an extension of GPU memory, effectively increasing the available memory capacity without requiring additional GPU memory hardware. The merge enables larger models to be processed by combining the memory resources of both processors while maintaining a simplified memory interface through the unified address space.
Solution Approach 2:
The patent implements multi-functionality by making CPU memory accessible to the GPU through the unified virtual memory system. The CPU memory serves dual purposes: it acts as the primary memory for the CPU processor and simultaneously functions as extended memory for the GPU. This universal usage of CPU memory for both CPU and GPU operations eliminates the need for dedicated GPU memory for all scenarios, reducing device complexity and cost while maintaining the ability to process large models.
3Speed
If direct loading from CPU memory to GPU cache is implemented, then data access speed is improved, but memory consistency management becomes more complex
Solution Approach 1:
The patent applies local quality by creating different memory access paths for different data types. Weight data that is pinned to CPU memory receives special treatment with direct loading paths to GPU cache, providing high-speed access optimized for this specific data type. Other data types continue to use the standard UVM page migration mechanism. This localized optimization for weight data achieves high data access speed without requiring complex consistency management for all memory operations, as only the pinned weight data path has the simplified direct loading behavior.
Data Source
AI summary
A method and apparatus for managing a unified virtual memory (UVM) are provided. The UVM is backed by a main processor memory and a coprocessor memory, and the method includes: checking properties of data blocks of the UVM used to execute a deep learning model; based on a first of the data blocks storing weight data of the deep learning model, storing the first data block in the main processor memory among the main processor memory and the coprocessor memory; and performing an operation of the deep learning model based on the first data block using a coprocessor while directly loading at least a portion of the first data block from the main processor memory into a cache memory of the coprocessor without migration of the first data block from the main processor memory to the coprocessor memory.