GPU Memory Footprint Estimation for ML Inference

Resolve Bottlenecks,
Find Innovative SolutionsGenerate Solutions

Solution Overview

Problem

Determining peak memory usage for large-scale machine-learning applications in GPU-embedded cloud containers is challenging due to the quadratic scaling of memory utilization with the number of sensors, leading to potential out-of-memory events and inefficient resource utilization, especially since Monte Carlo simulations are time-consuming and computationally intensive.

Innovation Solution

A system that computes an estimated memory footprint for an inferential model using formulae derived from execution parameters, such as the number of signals and desired precision, to determine the required number of GPUs and generate parallel execution code, allowing efficient memory usage in GPU-embedded cloud containers.

Engineering Contradictions & Design Principles

VSEngineering Contradiction Analysis

1Measurement precision

If Monte Carlo simulation is used to determine peak memory utilization, then measurement precision is improved, but loss of time and use of energy increase significantly

Engineering Contradiction:

Improvepeak memory utilization estimation accuracyVSAvoidtime consumed by Monte Carlo simulation

Core Design Contradiction:

Measurement precisionVSLoss of time

Solution Approach 1:

The patent replaces the computational simulation approach (Monte Carlo simulation) with an analytical mathematical formula approach. The formula directly calculates peak memory utilization based on execution parameters, eliminating the need for time-consuming iterative simulations while maintaining estimation accuracy.

Inventive Principle:

Principle #28Mechanics substitution (Replace mechanical system)

Solution Approach 2:

The patent transforms the problem from stochastic simulation to deterministic parameter calculation. By changing from Monte Carlo simulation to a direct formula based on execution parameters (number of signals, training vectors, observations, precision), the system achieves fast and accurate memory footprint estimation without the time cost of simulations.

Inventive Principle:

Principle #35Parameter changes

2Reliability

If conservative pre-allocation of RAM is used to prevent out-of-memory events, then reliability is improved, but loss of energy and resource utilization efficiency worsen

Engineering Contradiction:

Improveprevention of out-of-memory eventsVSAvoidenergy wasted due to memory resource underutilization

Core Design Contradiction:

ReliabilityVSLoss of energy

Solution Approach 1:

The patent implements a feedback mechanism where the system calculates the exact peak memory footprint using the analytical formula based on execution parameters, then allocates memory precisely according to this calculation. This eliminates the need for conservative over-allocation while ensuring sufficient memory is allocated to prevent out-of-memory events.

Inventive Principle:

Principle #23Feedback

Solution Approach 2:

The patent changes from conservative fixed memory allocation to dynamic parameter-based allocation. By using the analytical formula to calculate the exact memory footprint required based on execution parameters, the system allocates memory efficiently without waste, preventing both out-of-memory events and resource underutilization.

Inventive Principle:

Principle #35Parameter changes

3Adaptability or versatility

If VRAM is allocated between multiple GPUs, then adaptability is improved, but device complexity increases due to non-contiguous shareable memory

Engineering Contradiction:

Improveflexibility in GPU configurationVSAvoidcomplexity of VRAM allocation and parallel programming

Core Design Contradiction:

Adaptability or versatilityVSDevice complexity

Solution Approach 1:

The patent performs preliminary calculation of the total memory footprint required using the analytical formula before GPU allocation. This pre-calculation allows the system to determine the exact number of GPUs needed and allocate VRAM accordingly, simplifying the complexity of distributed memory allocation across multiple GPUs.

Inventive Principle:

Principle #10Preliminary action

Solution Approach 2:

The patent uses the analytical formula as a template or copy of the memory allocation strategy. By deriving the required memory footprint from the formula based on execution parameters, the system can efficiently allocate VRAM across multiple GPUs without complex real-time calculations, reducing programming complexity.

Inventive Principle:

Principle #26Copying

Data Source

PatentUS12073250B2Determining memory requirements for large-scale ml applications to facilitate execution in GPU-embedded cloud containers

Publication Date: 2024.08.27 ORACLE INT CORP

AI summary

We disclose a system that executes an inferential model in VRAM that is embedded in a set of graphics-processing units (GPUs). The system obtains execution parameters for the inferential model specifying: a number of signals, a number of training vectors, a number of observations and a desired data precision. It also obtains one or more formulae for computing memory usage for the inferential model based on the execution parameters. Next, the system uses the one or more formulae and the execution parameters to compute an estimated memory footprint for the inferential model. The system uses the estimated memory footprint to determine a required number of GPUs to execute the inferential model, and generates code for executing the inferential model in parallel while efficiently using available memory in the required number of GPUs. Finally, the system uses the generated code to execute the inferential model in the set of GPUs.