GPU Memory Partition Isolation for Error Containment
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Physical defects in memory components of graphics processing units (GPUs) can cause errors that impact multiple virtual servers running on a single physical server, affecting their operations and requiring effective isolation and migration of partitions to prevent cascading failures.
Innovation Solution
The implementation of hardware and software features on GPUs for partitioning, including a partition engine, direct memory access (DMA) support, video engine, and unique identifiers, along with error handlers to isolate and manage memory errors, ensuring that errors in one partition do not affect other partitions.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If memory errors are detected in a GPU partition, then the affected partition can be isolated and migrated, but the complexity of error handling and partition management increases
Solution Approach 1:
The GPU system is divided into multiple isolated partitions, each with its own memory space and error handling context. This segmentation allows errors to be contained within individual partitions without affecting other partitions, enabling reliable isolation while maintaining manageable complexity through modular error handling.
Solution Approach 2:
An intermediary error handling mechanism is introduced that sits between the GPU hardware and the virtual server software. This intermediary layer captures memory errors, determines the affected partition, and coordinates the isolation and migration process, thereby managing the complexity of error handling in a systematic way.
2Productivity
If multiple virtual servers run on a single physical server, then resource utilization increases, but errors in one server can impact other servers
Solution Approach 1:
The physical GPU server is segmented into multiple isolated partitions, with each virtual server assigned to a specific partition. This physical and logical segmentation prevents error propagation between servers while maintaining high resource utilization through efficient partition management and shared hardware resources.
Data Source
AI summary
Apparatuses, systems, and techniques to detect memory errors and isolate or migrate partitions on a parallel processing unit using an application programming interface to facilitate parallel computing, such as CUDA. In at least one embodiment, interrupts are intercepted and processed on a graphics processing unit indicating a memory error for one or more partitions, and a policy is applied to isolate that memory error from other partitions.