AI Model Co-Location Using Affinity Scores for Accelerator Throughput

Solution Overview

Problem

Existing methods for increasing the throughput of AI models by co-locating them on accelerators face inefficiencies due to resource bottlenecks and interference, leading to suboptimal utilization of limited resources.

Innovation Solution

A processor-based method analyzes computational characteristics of multiple models to determine an affinity score for co-location, considering memory capacity, bandwidth, and intermediate buffer usage, and co-locates models with high affinity scores to minimize resource interference and maximize utilization.

Engineering Contradictions & Design Principles

VSEngineering Contradiction Analysis

1Productivity

If multiple models are co-located on an accelerator to increase throughput, then the throughput of AI models is improved, but resource bottlenecks and interference occur leading to suboptimal utilization

Engineering Contradiction:

Improvethroughput of AI modelsVSAvoidresource utilization efficiency

Core Design Contradiction:

ProductivityVSDevice complexity

Solution Approach 1:

The system performs preliminary analysis of computational characteristics (memory capacity, bandwidth, intermediate buffer usage) and calculates affinity scores between model pairs before co-location. This preliminary assessment enables optimal model pairing decisions that prevent resource bottlenecks and interference, thereby maintaining high throughput while ensuring efficient resource utilization.

Inventive Principle:

Principle #10Preliminary action

Solution Approach 2:

The system changes the parameter of model selection by using affinity scores derived from computational characteristics such as memory capacity, bandwidth requirements, and intermediate buffer usage. By adjusting which models are co-located based on these parameters, the system optimizes both throughput and resource utilization efficiency simultaneously.

Inventive Principle:

Principle #35Parameter changes

2Productivity

If models with different resource requirements are co-located, then accelerator utilization is improved, but resource interference and bottlenecks increase

Engineering Contradiction:

Improveaccelerator utilizationVSAvoidresource interference

Core Design Contradiction:

ProductivityVSReliability

Solution Approach 1:

The system performs preliminary analysis of computational characteristics (memory capacity, bandwidth, intermediate buffer usage) and calculates affinity scores between model pairs before co-location. This preliminary assessment enables optimal model pairing decisions that prevent resource bottlenecks and interference, thereby maintaining high throughput while ensuring efficient resource utilization.

Inventive Principle:

Principle #10Preliminary action

Solution Approach 2:

The affinity score calculation provides feedback on resource compatibility between models. By analyzing computational characteristics and determining affinity scores, the system receives feedback on which model combinations will minimize resource interference, enabling informed co-location decisions that balance utilization and reliability.

Inventive Principle:

Principle #23Feedback

Data Source

PatentUS12602257B2Electronic device and operating method with model co-location

Publication Date: 2026.04.14 SAMSUNG ELECTRONICS CO LTD

AI summary

An electronic device to co-locating models and a method of operating the electronic device is provided. The electronic device includes one or more of processors configured to analyze computational characteristics in response to a plurality of models being located to an accelerator, determine an affinity representing a utilization of the accelerator in response to two models among the plurality of models being co-located based on the computational characteristics of the plurality of models, and co-locate the two models among the plurality of models to the accelerator based on the affinity.