Hierarchical Scratchpad Memory for AI Accelerator Bandwidth Bottlenecks

Resolve Bottlenecks,
Find Innovative SolutionsGenerate Solutions

Solution Overview

Problem

Current deep learning algorithms face memory bandwidth limitations, leading to suboptimal performance in AI workloads, despite the use of high-bandwidth memories, due to inefficient data structure placement and memory allocation in Deep Neural Networks (DNNs).

Innovation Solution

The implementation of an optimized hierarchical scratchpad memory architecture, where data structures are strategically placed across multiple scratchpad levels and external memory based on their reuse, duration, and operational requirements, with an additional scratchpad level positioned between the on-chip memory and external memory to enhance data reuse and reduce memory bandwidth constraints.

Engineering Contradictions & Design Principles

VSEngineering Contradiction Analysis

1Quantity of substance

If data structures are placed in external memory, then memory capacity is sufficient, but memory bandwidth limitations cause suboptimal performance

Engineering Contradiction:

Improvememory capacityVSAvoidAI accelerator core utilization

Core Design Contradiction:

Quantity of substanceVSProductivity

Solution Approach 1:

The memory system is segmented into multiple scratchpad levels (first scratchpad memory and second scratchpad memory) positioned between external memory and AI accelerator cores. This segmentation allows data to be divided and placed strategically across different memory levels based on access patterns, with frequently accessed data in closer scratchpads and less frequent data in external memory, thereby resolving the bandwidth-capacity tradeoff.

Inventive Principle:

Principle #1Segmentation

Solution Approach 2:

The patent introduces an additional hierarchical dimension to the memory architecture by adding a second scratchpad memory level between external memory and the first scratchpad. This creates a multi-level hierarchy that adds temporal and spatial dimensions to data placement, enabling optimized data reuse without increasing external bandwidth requirements.

Inventive Principle:

Principle #17Another dimension (Dimensionality change)

2Quantity of substance

If high-bandwidth external memory is used, then memory capacity increases, but memory bandwidth limitations still constrain AI workload performance

Engineering Contradiction:

Improvememory capacityVSAvoidmemory bandwidth

Core Design Contradiction:

Quantity of substanceVSSpeed

Solution Approach 1:

The first and second scratchpad memories act as intermediary buffers between external memory and AI accelerator cores. These intermediaries cache data locally, reducing the need for frequent high-speed external memory accesses. The scratchpads mediate data transfer by holding frequently accessed data closer to the compute units, thereby mitigating external bandwidth limitations while maintaining large memory capacity.

Inventive Principle:

Principle #24Intermediary (Mediator)

3Speed

If data is placed closer to AI accelerator cores, then access speed improves, but memory capacity is limited

Engineering Contradiction:

Improvedata access speedVSAvoidmemory capacity

Core Design Contradiction:

SpeedVSQuantity of substance

Solution Approach 1:

Inventive Principle:

Principle #1Segmentation

Solution Approach 2:

The memory architecture implements a nested hierarchy where the second scratchpad memory is positioned between external memory and the first scratchpad memory. This nested structure allows smaller, faster memory levels to be embedded within the larger memory system, enabling fast access to critical data while maintaining adequate capacity in external memory for less frequently accessed data.

Inventive Principle:

Principle #7Nested doll (Nesting)

Data Source

PatentUS11429524B2Optimized hierarchical scratchpads for enhanced artificial intelligence accelerator core utilization

Publication Date: 2022.08.30 INTERNATIONAL BUSINESS MACHINE CORPORATION

AI summary

Various embodiments are provided for optimized placement of data structures in a hierarchy of memory in a computing environment. One or more data structures may be placed in a first scratchpad memory, a second scratchpad memory, an external memory, or a combination thereof in the hierarchy of memory according to a total memory capacity and bandwidth, a level of reuse of the one or more data structures, a number of operations that use each of the one or more data structures, a required duration each the one or more data structures are required to be placed a first scratchpad or a second scratchpad, and characteristics of those of the one or more data structures competing for placement in the hierarchy of memory that are able to co-exist at a same time step. The second scratchpad memory is positioned between the external memory and the first scratchpad memory at one or more intermediary layers.