GPU Multistream Memory Reuse for Fragmentation-Aware Allocation
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
The limited capacity of graphics memories in GPUs restricts the efficiency of large-scale computing tasks, such as deep learning, despite their high bandwidth, necessitating improved methods to utilize memory resources effectively.
Innovation Solution
A graphics memory reuse method and apparatus based on GPU multistream concurrency, where at least two GPU streams are concurrently executed, allowing for the determination and allocation of reusable graphics memory blocks from a pool, with modes like default and multi-stream reuse to optimize memory usage.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Quantity of substance
If graphics memory capacity is increased to support large-scale computing tasks, then the ability to handle large models and samples improves, but the cost and complexity of the GPU system increases
Solution Approach 1:
The patent merges multiple small graphics memory blocks from different streams into a unified graphics memory pool, allowing efficient utilization of fragmented memory resources. This combining approach enables large-scale computing tasks to be supported without requiring a single large memory capacity, thus avoiding increased hardware complexity and cost.
Solution Approach 2:
The graphics memory pool serves multiple functions: it acts as a shared resource for multiple GPU streams, provides both allocation and reuse capabilities, and supports various data structures (buffers, tensors, activations). This multi-functionality allows the system to handle large-scale computing tasks using a moderate total memory capacity, reducing the need for expensive high-capacity memory.
2Reliability
If graphics memory is allocated to each GPU stream independently, then stream isolation and reliability are maintained, but memory utilization efficiency deteriorates due to fragmentation and waste
Solution Approach 1:
The patent segments the graphics memory into multiple small blocks that can be independently allocated and reused. Each stream maintains logical isolation through its own allocation view, while physically sharing the segmented memory pool. This segmentation enables high utilization efficiency by allowing fine-grained memory reuse without compromising stream reliability.
Solution Approach 2:
The patent introduces a graphics memory management unit as an intermediary between GPU streams and the physical memory blocks. This mediator manages allocation, tracking, and reuse of memory blocks, maintaining stream isolation through virtualization while enabling efficient reuse by coordinating allocations across streams. The intermediary resolves the conflict between isolation and efficiency.
3Productivity
If a graphics memory pool is introduced to enable reuse, then memory utilization efficiency improves, but the complexity of memory management increases
Solution Approach 1:
The graphics memory management unit implements self-service mechanisms by automatically tracking memory block states, identifying reusable blocks, and allocating them without extensive external intervention. The system maintains its own allocation tables and state information, enabling efficient reuse while keeping management complexity contained within the dedicated management unit rather than propagating it throughout the entire system.
Data Source
AI summary
Embodiments of this specification provide graphics memory reuse methods and apparatuses based on GPU multistream concurrency. In an implementation of a default stream reuse mode, a method includes determining, based on (1) a released graphics memory corresponding to a current GPU stream that comprises a GPU instruction to which a graphics memory is to be allocated and (2) whether the current GPU stream is a default stream, whether a candidate reusable graphics memory block exists in a graphics memory pool for storing a released graphics memory block. If the candidate reusable graphics memory block exists, determining, from the candidate reusable graphics memory block, a graphics memory block to be allocated to the GPU instruction.