Unified Memory Allocation for Neural Network Inference Latency

Resolve Bottlenecks,
Find Innovative SolutionsGenerate Solutions

Solution Overview

Problem

Existing neural network accelerators face latency issues due to the time-consuming processes of writing input data to memory and reading output data from memory, particularly when handling streaming inputs.

Innovation Solution

An integrated circuit (IC) is designed with a unified memory accessible by input processing circuits, neural network computation fabric, and a microprocessor, allowing simultaneous execution of neural networks for different inputs while optimizing memory access.

Engineering Contradictions & Design Principles

VSEngineering Contradiction Analysis

1Loss of time

If a typical neural network accelerator writes input data to memory and reads output data from memory sequentially, then the memory access process is simple, but the latency increases significantly

Engineering Contradiction:

Improvememory access latencyVSAvoidmemory access mechanism complexity

Core Design Contradiction:

Loss of timeVSDevice complexity

Solution Approach 1:

The patent applies preliminary action by pre-loading input data into dedicated input memory buffers before the neural network computation starts. The input processing circuit writes input data to memory banks in advance, allowing the computation fabric to immediately read from these pre-prepared buffers without waiting for sequential memory writes during computation, thereby reducing memory access latency.

Inventive Principle:

Principle #10Preliminary action

Solution Approach 2:

The patent segments the memory system into different memory banks with specific functions: input memory banks for storing input data, output memory banks for storing computation results, and weight memory banks for storing neural network parameters. This segmentation allows simultaneous independent operations in different memory regions, enabling the input processing circuit to write to input memory while the computation fabric reads from weight memory without interference, thus reducing overall memory access latency.

Inventive Principle:

Principle #1Segmentation

2Speed

If the neural network computation fabric executes networks for streaming inputs sequentially, then the execution process is simple, but the processing speed decreases

Engineering Contradiction:

Improveneural network execution speedVSAvoidmemory bank allocation and switching complexity

Core Design Contradiction:

SpeedVSDevice complexity

Solution Approach 1:

The patent implements dynamic memory bank allocation where the system can switch between different memory banks based on the current operation phase. The microprocessor controls the allocation of memory banks to different circuits based on whether the system is in data loading phase or computation phase, allowing flexible adaptation to different operational requirements and enabling faster processing of streaming inputs.

Inventive Principle:

Principle #15Dynamics

Solution Approach 2:

The input processing circuit performs preliminary action by writing input data to dedicated input memory banks before the computation fabric needs to process the data. This pre-preparation allows the computation fabric to immediately begin processing without waiting for sequential memory transfers, significantly increasing the execution speed for streaming inputs.

Inventive Principle:

Principle #10Preliminary action

3Productivity

If the unified memory is shared by multiple circuits, then the memory utilization is efficient, but the memory access conflicts increase

Engineering Contradiction:

Improvememory utilization efficiencyVSAvoidmemory access consistency

Core Design Contradiction:

ProductivityVSReliability

Solution Approach 1:

The patent segments the unified memory into different memory banks with dedicated access rights: input memory banks are accessible to the input processing circuit, output memory banks to the computation fabric, and weight memory banks to both. This segmentation maintains efficient memory utilization while preventing access conflicts through clear delineation of which circuits can access which memory regions simultaneously.

Inventive Principle:

Principle #1Segmentation

Solution Approach 2:

The microprocessor acts as an intermediary controller that manages memory bank allocation and access coordination between different circuits. It controls the allocation of memory banks to different circuits based on operational requirements, mediating between the input processing circuit, computation fabric, and any other circuits to ensure consistent and conflict-free memory access while maintaining high utilization efficiency.

Inventive Principle:

Principle #24Intermediary (Mediator)

Data Source

PatentUS12217160B1Allocating blocks of unified memory for integrated circuit executing neural network

Publication Date: 2025.02.04 AMAZON COM SERVICES LLC

AI summary

Some embodiments provide a method that receives a specification of a neural network for execution by an integrated circuit. The integrated circuit includes a neural network inference circuit for executing the neural network to generate an output based on an input, an input processing circuit for providing the input to the neural network inference circuit, a microprocessor circuit for controlling the neural network inference circuit and the input processing circuit, and a unified memory accessible by the microprocessor circuit, the neural network inference circuit, and the input processing circuit. The method determines usage of the unified memory by the neural network inference circuit while executing the neural network. Based on the determined usage by the neural network inference circuit, the method allocates portions of the unified memory to the microprocessor circuit and input processing circuit.