Operator-Aware PIM Offloading for AI Inference

Resolve Bottlenecks,
Find Innovative SolutionsGenerate Solutions

Solution Overview

Problem

Existing artificial intelligence models face challenges in efficiently improving inference speed due to the limitations of processing-in-memory (PIM) technology, which is underdeveloped and lacks appropriate methods for scheduling offloading based on the type of computation performed, leading to varying performance outcomes.

Innovation Solution

A method, device, and computer program for processing-in-memory computation offloading that determines the usefulness of offloading computations to PIM based on the type of operator, operand size, and optimal operand size, considering initial and generation phases, and calculating offloading benefits and overheads to optimize inference performance.

Engineering Contradictions & Design Principles

VSEngineering Contradiction Analysis

1Productivity

If processing-in-memory (PIM) technology is used for computation offloading, then computational efficiency is improved, but device complexity increases

Engineering Contradiction:

Improvecomputational efficiencyVSAvoiddevice complexity

Core Design Contradiction:

ProductivityVSDevice complexity

Solution Approach 1:

The system segments computations into different types (GEMV, GEMM, and other operators) and selectively offloads only suitable computations to PIM devices. The computation is divided into initial phase and generation phase, with different offloading strategies applied to each phase, thereby improving computational efficiency while managing device complexity through selective offloading.

Inventive Principle:

Principle #1Segmentation

Solution Approach 2:

The offloading decision is made dynamically based on the type of computation being performed. The system adaptively determines whether to offload each computation to PIM or keep it on the host processor, allowing the system to optimize computational efficiency for suitable operations while avoiding the complexity overhead for operations that do not benefit from PIM acceleration.

Inventive Principle:

Principle #15Dynamics

2Productivity

If computation offloading to PIM is performed for all computation types, then productivity is improved, but the system becomes less adaptable to different computation scenarios

Engineering Contradiction:

Improveinference performanceVSAvoidadaptability to computation types

Core Design Contradiction:

ProductivityVSAdaptability or versatility

Solution Approach 1:

The system applies different offloading strategies to different computation types and phases. GEMV and GEMM computations in the generation phase are offloaded to PIM for improved performance, while other computation types and the initial phase are processed on the host processor. This localized approach ensures adaptability to different computation scenarios while maintaining high productivity for suitable operations.

Inventive Principle:

Principle #3Local quality

Solution Approach 2:

The system changes the offloading parameter (whether to offload to PIM) based on the computation type and phase. By dynamically adjusting this parameter according to the specific computation being performed, the system achieves both high productivity for PIM-suited operations and adaptability to various computation scenarios.

Inventive Principle:

Principle #35Parameter changes

3Ease of operation

If PIM offloading is implemented without selective determination, then ease of operation is improved, but loss of time occurs due to inefficient offloading decisions

Engineering Contradiction:

Improveease of offloading implementationVSAvoidtime loss from inefficient offloading

Core Design Contradiction:

Ease of operationVSLoss of time

Solution Approach 1:

The system performs preliminary analysis of the computation type and phase before making offloading decisions. By categorizing computations into GEMV, GEMM, and other operators, and identifying whether they belong to the initial or generation phase, the system makes informed offloading decisions that avoid time loss from inefficient PIM usage while maintaining ease of operation through automated classification.

Inventive Principle:

Principle #10Preliminary action

Solution Approach 2:

The system uses feedback from computation type analysis to guide offloading decisions. By continuously evaluating the type of computation being performed and adjusting offloading decisions accordingly, the system minimizes time loss from inefficient offloading while maintaining ease of operation through automated, adaptive decision-making.

Inventive Principle:

Principle #23Feedback

Data Source

PatentUS20250335348A1Method, device, system, and computer program for processing-in-memory computation offloading for improving inference performance of artificial intelligence model

Publication Date: 2025.10.30 SAMSUNG SDS CO LTD

AI summary

The present disclosure relates to a method, a device, a system, and a computer program for processing-in-memory computation offloading for improving the inference performance of an artificial intelligence model. More specifically, the present disclosure provides a method for performing processing-in-memory (PIM) offloading by using a computing device, the method including: collecting information about a first computation to be processed, the first computation including an operator and at least one operand; determining the usefulness of offloading the first computation, based on the information about the first computation and the optimal operand size of a processing-in-memory (PIM); and offloading the first computation, based on the determination.