3D Memory-Logic NPU Layout for Low-Latency Neural Computation
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Current neural network computation methods face inefficiencies due to high power consumption, latency, and inflexibility in handling diverse AI workloads, particularly in resource-constrained environments like mobile devices and IoT applications, and existing solutions like GPUs, compute-in-memory, and memory-in-compute face challenges in scalability, flexibility, and thermal management.
Innovation Solution
An integrated 3D system with a memory die stacked over a logic die, utilizing DRAM for dynamic data storage and TSVs for high-bandwidth, low-latency connections, reducing data transfer and enabling adaptable, scalable, and efficient computations.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Adaptability or versatility
If data is transferred between separate memory and compute units, then computational flexibility is improved, but power consumption increases and latency increases
Solution Approach 1:
The patent merges memory and compute units into a single integrated neural processing unit (NPU) architecture. Memory blocks are directly coupled to compute blocks within the same die, eliminating the need for separate memory chips and reducing data transfer distances. This integration maintains computational flexibility while significantly reducing power consumption associated with data movement between discrete components.
Solution Approach 2:
The patent implements a three-dimensional stacked architecture where memory blocks and compute blocks are arranged in multiple layers vertically. This 3D organization allows memory and compute units to be closely coupled in three-dimensional space, reducing lateral data transfer distances and improving bandwidth while maintaining architectural flexibility for various neural network operations.
2Adaptability or versatility
If data is transferred between separate memory and compute units, then computational flexibility is improved, but latency increases
Solution Approach 1:
The integration of memory and compute units within the same NPU die reduces data transfer latency by eliminating external memory interface overhead and reducing physical transmission distances. Memory blocks are directly accessible by compute blocks through short on-chip interconnects, enabling faster data access while maintaining support for various neural network operations.
Solution Approach 2:
The 3D stacked architecture positions memory and compute blocks in vertically adjacent layers, minimizing data transfer paths through short vertical interconnects. This spatial arrangement dramatically reduces latency compared to planar 2D layouts or separate chip configurations, while preserving architectural flexibility through configurable memory-compute mappings.
3Use of energy by moving object
If compute-in-memory architecture is used, then power consumption is reduced, but scalability and flexibility are limited
Solution Approach 1:
The NPU architecture implements universal compute blocks that can perform multiple neural network operations including matrix multiplications, convolutions, and activation functions. The same compute blocks can be configured for different operations through programmable control, providing flexibility across various neural network models while maintaining the low-power benefits of integrated memory-compute architecture.
Solution Approach 2:
The patent employs dynamically reconfigurable interconnects and memory block assignments that can adapt to different computational workloads. Memory blocks can be dynamically allocated to different compute blocks based on operational requirements, and data flow paths can be reconfigured to optimize performance for specific neural network layers or operations, enabling scalability across different model sizes and types.
4Productivity
If memory and compute are integrated on the same die, then data transfer is minimized, but thermal management becomes more challenging
Solution Approach 1:
The NPU die is segmented into multiple independent memory blocks and compute blocks distributed across different regions. This segmentation allows heat to be dispersed across the die surface rather than concentrated in a single location, improving thermal management while maintaining the benefits of integrated memory-compute architecture for efficient data transfer.
Data Source
AI summary
An integrated circuit (IC) device may implement a deep neural network (DNN). The IC device may be a three-dimensional (3D) integrated system that includes a memory die and logic die. The memory die may include memory blocks, such as sequential random-access memory blocks or a sequential read-only memory blocks. The logic die may include an interface unit, a vector operation unit, compute units (e.g., multiply-accumulate units), and an interconnect fabric with adders. The interface unit may receive the input of the DNN and transfer the input to the vector operation unit. The vector operation unit may perform one or more vector operations of the DNN based on the input. The compute units and adders may perform matrix multiplication operations of the DNN based on the vector operation unit's output. Each memory block may be coupled with a compute unit through a via.