Slab Memory Management for Machine Learning Training
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Current machine learning training systems face inefficiencies due to separated computing and storage units, leading to performance and energy inefficiencies, particularly with limited GPU DRAM size and poor memory management schemes that increase overhead in machine learning operations.
Innovation Solution
A method and system for slab-based memory management that allocates memory segments efficiently by identifying the least recently used segments and migrating them to secondary memory, allowing for optimized calculations and storage in machine learning operations, specifically utilizing a user-level process to manage memory allocations and migrations within a GPU and secondary storage.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Ease of operation
If heterogeneous memory management schemes with page granularity are used for GPUs, then memory can be managed with standard operating system mechanisms, but memory management overhead increases and performance deteriorates in machine learning training operations
Solution Approach 1:
The patent segments memory management by creating separate management mechanisms for different data types in machine learning workloads. It implements type-specific memory pools (e.g., activation pools, weight pools) that are managed independently from general page-based memory management, allowing optimized handling of ML-specific memory access patterns while maintaining compatibility with standard OS mechanisms.
Solution Approach 2:
The patent changes the granularity parameter of memory management from page-level (standard OS approach) to object-level or block-level management tailored for machine learning data structures. This allows memory to be allocated and freed in larger, more efficient units that match ML workload requirements, reducing the frequency of memory management operations and improving performance.
2Use of energy by stationary object
If GPU DRAM size is reduced due to power and energy constraints in SSDs, then in-storage machine learning training becomes feasible, but memory capacity for training operations is limited
Solution Approach 1:
The patent implements a nested memory hierarchy where small, fast GPU DRAM is nested within the SSD, which itself is nested within the larger system memory hierarchy. This allows the limited GPU DRAM to be used for frequently accessed data while less frequently accessed data resides in the SSD's storage memory, creating a tiered structure that maximizes the utility of limited fast memory while providing access to larger capacity through slower storage layers.
Solution Approach 2:
The patent extends the memory space available to the GPU by adding a spatial dimension through the SSD storage memory. Instead of being constrained to a single memory layer, the system creates a multi-layer memory architecture where data can be moved between GPU DRAM, SSD storage, and system memory, effectively increasing the addressable memory space without increasing the size of the fast GPU DRAM.
Data Source
AI summary
A system and method for machine learning, with a processing circuit executing an operating system, the processing circuit being connected to a first memory and to a second memory. In some embodiments, the method includes: requesting, by a user level process, from the operating system, a first memory allocation from the first memory, the first memory allocation including a plurality of first segments, a first segment of the plurality of first segments having a size sufficient to store a data object of a first kind; determining, by the user level process, that a result of a first calculation will be a data object of the first kind; and in response to determining that the result of the first calculation will be a data object of the first kind: determining, by the user level process, that none of the first segments are unused.