Multi-Crossbar Topology for NPU Memory Data Transmission

Resolve Bottlenecks,
Find Innovative SolutionsGenerate Solutions

Solution Overview

Problem

Multi-neural processing unit (NPU) systems face challenges in effective data transmission between NPUs and scratchpad memory (SPM) due to high hardware complexity and scalability issues, leading to performance degradation and increased leakage power.

Innovation Solution

The implementation of a scalable multi-crossbar topology that groups processors and memories, using fully connected crossbars within groups and 2x2 crossbars to connect neighboring groups, reducing hardware complexity and memory contention while maintaining performance.

Engineering Contradictions & Design Principles

VSEngineering Contradiction Analysis

1Productivity

If a multi-NPU system uses scratchpad memory (SPM) instead of cache memory, then the system can perform large-scale deep learning networks, but effective connection between NPU and SPM is lacking

Engineering Contradiction:

Improvedeep learning operation capabilityVSAvoidconnection effectiveness

Core Design Contradiction:

ProductivityVSDevice complexity

Solution Approach 1:

The system divides NPUs and SPMs into multiple groups, with each group having dedicated crossbar switches for data transmission. This segmentation reduces contention and improves connection effectiveness between NPUs and SPMs while maintaining large-scale deep learning capability.

Inventive Principle:

Principle #1Segmentation

Solution Approach 2:

Crossbar switches are introduced as intermediary components between NPUs and SPMs to facilitate efficient data transmission. The crossbar switches act as mediators that manage data flow and improve connection effectiveness without requiring direct NPU-SPM connections.

Inventive Principle:

Principle #24Intermediary (Mediator)

2Productivity

If the number of NPUs and SPMs is increased to perform deep learning applications, then processing capability is improved, but hardware complexity increases

Engineering Contradiction:

Improvedeep learning processing capabilityVSAvoidhardware complexity

Core Design Contradiction:

ProductivityVSDevice complexity

Solution Approach 1:

The system segments NPUs and SPMs into groups with dedicated crossbar switches, allowing the hardware to scale to support more NPUs and SPMs without proportionally increasing overall complexity. Each group operates semi-independently, reducing the complexity burden of scaling.

Inventive Principle:

Principle #1Segmentation

Solution Approach 2:

The crossbar switch design provides universal data transmission capability that can accommodate varying numbers of NPUs and SPMs. The same crossbar architecture serves multiple functions: intra-group data transmission, inter-group communication, and memory access, reducing the need for specialized components as the system scales.

Inventive Principle:

Principle #6Universality (Multi-functionality)

3Speed

If fully connected crossbars are used between all NPUs and SPMs, then data transmission efficiency is improved, but hardware complexity and leakage power increase

Engineering Contradiction:

Improvedata transmission efficiencyVSAvoidcrossbar complexity

Core Design Contradiction:

SpeedVSDevice complexity

Solution Approach 1:

Instead of implementing a single fully connected crossbar across all NPUs and SPMs, the system segments the crossbar into multiple smaller crossbars, each serving a specific group. This maintains data transmission efficiency within groups while reducing the complexity and leakage power of any single crossbar component.

Inventive Principle:

Principle #1Segmentation

Solution Approach 2:

The system implements full connectivity locally within each NPU-SPM group through dedicated crossbars, ensuring high data transmission efficiency for local operations. Inter-group communication is handled separately, allowing each crossbar to be optimized for its specific group size, thereby reducing overall hardware complexity and leakage power.

Inventive Principle:

Principle #3Local quality

4Adaptability or versatility

If more crossbars are added to connect all NPUs and SPMs, then data transmission flexibility is improved, but hardware complexity and leakage power increase

Engineering Contradiction:

Improvedata transmission flexibilityVSAvoidleakage power

Core Design Contradiction:

Adaptability or versatilityVSLoss of energy

Solution Approach 1:

The system divides the crossbar infrastructure into multiple smaller crossbars, each handling specific groups of NPUs and SPMs. This segmentation provides sufficient data transmission flexibility for deep learning operations while reducing the number of active crossbar components at any given time, thereby lowering leakage power compared to a single large fully connected crossbar.

Inventive Principle:

Principle #1Segmentation

Data Source

PatentUS20240211744A1Apparatus and method with multiple neural processing units for neural network operation

Publication Date: 2024.06.27 SAMSUNG ELECTRONICS CO LTD

AI summary

An apparatus includes: memories storing data to perform a neural network operation; processors to generate a neural network operation result by performing a neural network operation by reading the data; and crossbars processing data transmission between the processors and the memories, wherein the crossbars include: a first crossbar of a first group processing data transmission between a first group of the processors and a first group of the memories, a second crossbar of a second group processing data transmission between a second group of the processors and a second group of the memories, wherein the first group of processors does not include any processors that are in the second group of processors and wherein the first group of memories does not include any memories that are in the second group of memories, and a third crossbar connecting the first crossbar to the second crossbar.