SoC ML Model Memory Partitioning for Low-Power Inference

Resolve Bottlenecks,
Find Innovative SolutionsGenerate Solutions

Solution Overview

Problem

Existing machine learning models consume significant power and require frequent data transfers between volatile and non-volatile memory, leading to inefficiencies and potential degradation of non-volatile memory due to excessive write cycles.

Innovation Solution

A system-on-chip (SoC) design that incorporates both flash memory and RAM, where fixed weight regions of the ML model are stored in flash memory and dynamic regions are stored in RAM, reducing power consumption and minimizing memory transfers by using a write-permission flag and direct access from flash memory.

Engineering Contradictions & Design Principles

VSEngineering Contradiction Analysis

1Speed

If machine learning models are stored entirely in volatile memory (RAM) for fast access, then inference speed is improved, but power consumption increases and data loss occurs when power is lost

Engineering Contradiction:

Improveinference speedVSAvoidpower consumption

Core Design Contradiction:

SpeedVSUse of energy by moving object

Solution Approach 1:

The patent divides the ML model storage into two segments: frequently accessed model weights are stored in volatile memory (RAM) for fast access during inference, while less frequently accessed or backup copies are stored in non-volatile memory (flash memory). This segmentation allows the system to balance between speed and power consumption by loading only necessary portions into RAM.

Inventive Principle:

Principle #1Segmentation

Solution Approach 2:

The patent applies local quality by storing different types of data in different memory locations with appropriate characteristics. Critical model parameters that require frequent access are placed in high-speed volatile memory, while model checkpoints and less frequently accessed data are stored in non-volatile memory, optimizing both speed and energy efficiency for each data type.

Inventive Principle:

Principle #3Local quality

2Use of energy by moving object

If machine learning models are stored entirely in non-volatile memory (flash memory) to save power, then power consumption is reduced, but inference speed decreases and memory transfer times increase

Engineering Contradiction:

Improvepower consumptionVSAvoidinference speed

Core Design Contradiction:

Use of energy by moving objectVSSpeed

Solution Approach 1:

The patent implements preliminary action by pre-loading essential model weights and parameters into volatile memory (RAM) during system initialization or idle periods when power consumption is less critical. This allows the frequently accessed data to be readily available in fast memory during actual inference operations, reducing transfer times while maintaining overall power efficiency.

Inventive Principle:

Principle #10Preliminary action

3Ease of operation

If frequent data transfers between volatile and non-volatile memory are performed, then data accessibility is improved, but non-volatile memory degradation accelerates due to excessive write cycles

Engineering Contradiction:

Improvedata accessibilityVSAvoidnon-volatile memory lifespan

Core Design Contradiction:

Ease of operationVSReliability

Solution Approach 1:

The patent extracts the write-intensive operations from the regular inference workflow by maintaining model weights in volatile memory (RAM) during inference operations. This eliminates the need for frequent write cycles to non-volatile memory, as the model parameters remain stationary in RAM during processing, thereby preserving flash memory reliability while maintaining data accessibility.

Inventive Principle:

Principle #2Taking out (Extraction)

Solution Approach 2:

The patent uses copying by creating and maintaining copies of model data in both volatile and non-volatile memory. The primary working copy resides in RAM for fast access during inference, while a backup or archival copy exists in flash memory. This copying strategy allows the system to access data efficiently from RAM without requiring frequent writes to flash memory, thus protecting it from degradation.

Inventive Principle:

Principle #26Copying

4Measurement precision

If entire machine learning models are loaded into volatile memory for processing, then inference accuracy is maintained, but memory bandwidth requirements increase and cost increases

Engineering Contradiction:

Improveinference accuracyVSAvoidmemory bandwidth

Core Design Contradiction:

Measurement precisionVSQuantity of substance

Solution Approach 1:

The patent applies partial action by loading only the essential model weights and parameters required for inference into volatile memory (RAM), rather than loading the entire model including all auxiliary data and less frequently accessed parameters. This selective loading approach maintains inference accuracy for the critical parameters while significantly reducing the memory bandwidth requirements and volatile memory capacity needs.

Inventive Principle:

Principle #16Partial or excessive action

Data Source

PatentUS20250378043A1System-on-chip having non-volatile memory storing a machine learning model and performing inference using the model

Publication Date: 2025.12.11 MERCEDES BENZ GROUP AG

AI summary

A computing device, such as a system-on-chip, can include non-volatile memory and volatile memory. The computing device can further include processing resources that read a set of values of the ML model and apply the set of values of the ML model to input data. Based on applying the set of values to the input data, the processing resources can determine regions of the ML model having fixed values and regions of the ML model having non-fixed values. Upon making this determination, the processing resources can migrate or store the regions of the ML model having non-fixed values in the volatile memory of the computing device.