3D Neural Inference Architecture for Memory Bandwidth Scaling
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing neural inference processing architectures face limitations in memory capacity, network size, and energy efficiency due to their physical layout, which restricts the scalability and performance of neural network computations.
Innovation Solution
Implementing a 3D neural inference processing unit architecture using silicon 3D die and wafer stacking, with through-silicon vias for inter-tier connectivity, and a 2D global weight distribution bus to increase bandwidth and memory capacity, allowing for scalable and efficient neural network computations.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Quantity of substance
If a 2D neural inference processing architecture is used, then the physical footprint is smaller, but the memory capacity and bandwidth are limited
Solution Approach 1:
The patent transitions from a 2D architecture to a 3D architecture by stacking multiple processing tiers vertically. This dimensional change allows memory capacity and computational resources to scale upward in the third dimension rather than outward in the plane, thereby increasing memory capacity and bandwidth without proportionally increasing the physical footprint area.
Solution Approach 2:
The patent implements a hierarchical nested structure where processing cores are organized into groups, with multiple tiers stacked and interconnected. Each tier contains processing units that are nested within the overall 3D structure, allowing dense packing of computational resources and memory structures that increase capacity without linearly increasing footprint.
2Loss of energy
If wire length is reduced in the 3D architecture, then energy consumption decreases, but inter-tier connectivity complexity increases
Solution Approach 1:
By moving connectivity resources to the vertical dimension through stacked tiers and through-silicon vias, the patent reduces the horizontal wire length between memory and processing units. This dimensional reorganization decreases energy consumption associated with long wire transmissions while the inter-tier connectivity complexity is managed through standardized vertical interconnect structures.
Solution Approach 2:
The patent introduces through-silicon vias and inter-tier interconnect structures as intermediary elements that facilitate communication between stacked tiers. These intermediaries provide standardized, high-density vertical connectivity pathways that reduce the complexity of direct long-range horizontal connections while enabling efficient data transfer with lower energy consumption.
3Productivity
If more processing units are added to increase parallelism, then computational efficiency improves, but memory capacity and bandwidth become bottlenecks
Solution Approach 1:
The patent merges processing units and memory structures into an integrated 3D architecture where multiple processing tiers are closely coupled with memory resources. This merging creates high-bandwidth interconnects between processing units and memory, allowing computational efficiency to scale with parallelism without being bottlenecked by memory bandwidth limitations.
Solution Approach 2:
By organizing memory and processing units in vertically stacked tiers, the patent increases the density of interconnects in the vertical dimension. This allows each processing unit to access memory resources with higher bandwidth through short vertical pathways, enabling increased parallelism without memory bandwidth becoming a limiting factor.
Data Source
Figure 1
Figure 2
Figure 3
AI summary
Three-dimensional neural inference processing units are provided. A first tier comprises a plurality of neural cores. Each core comprises a neural computation unit. The neural computation unit is adapted to apply a plurality of synaptic weights to a plurality of input activations to produce a plurality of output activations. A second tier comprises a first neural network model memory adapted to store the plurality of synaptic weights. A communication network is operatively coupled to the first neural network model memory and to each of the plurality of neural cores, and adapted to provide the synaptic weights from the first neural network model memory to each of the plurality of neural cores.