3D Stacked Memory ASIC for Neural Network Data Access
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Complex artificial neural networks require fast and efficient access to high-performance memory for both training and inference, but existing architectures often suffer from inefficiencies in connection distances and communication between processing elements and memory, limiting performance and efficiency.
Innovation Solution
The proposed solution involves an application-specific integrated circuit (ASIC) with an architecture that minimizes connection distances between processing elements and memory, supporting concurrent forward and back propagation through a systolic array architecture with interconnected processing tiles and staging buffers, enabling efficient communication and data flow for neural network operations.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Speed
If existing memory architectures are used with neural networks, then memory capacity is provided, but connection distances between processing elements and memory are long causing inefficiency
Solution Approach 1:
The patent transitions from planar 2D memory architecture to 3D stacked memory architecture, adding a vertical dimension to the system. Memory stacks are positioned directly above processing elements, creating short vertical connections through through-silicon vias (TSVs). This dimensional change reduces connection distance dramatically while maintaining high memory capacity, directly resolving the contradiction between fast access and long connection distances.
2Productivity
If complex neural networks are implemented, then computational capability increases, but communication efficiency between processing elements and memory decreases
Solution Approach 1:
The system is segmented into multiple processing elements (PEs) arranged in a mesh network, each with dedicated local memory stacks. This segmentation allows parallel processing of neural network computations while each PE communicates only with nearby memory, reducing communication overhead and energy loss. The mesh network further segments communication paths, enabling efficient data exchange between PEs without bottlenecks.
Solution Approach 2:
By stacking memory vertically above each PE and using TSVs for inter-layer communication, the architecture creates short vertical communication paths. This 3D arrangement enables complex neural network computations to proceed efficiently, as each PE can access its required memory data through short vertical connections rather than long horizontal traces, significantly improving communication efficiency and reducing energy loss.
3Productivity
If conventional architectures are used, then system simplicity is maintained, but idle time increases reducing overall efficiency
Solution Approach 1:
The architecture pre-positions multiple memory stacks directly above each processing element, so that all required data for computation is immediately accessible when needed. This preliminary arrangement eliminates idle time by ensuring that processing elements never wait for data fetches from distant memory locations. The systolic array architecture further enables continuous data flow through the network, keeping all elements actively computing without idle periods.
Data Source
AI summary
An application-specific integrated circuit for an artificial neural network is integrated with a high-bandwidth memory. The neural network includes a systolic array of interconnected processing elements, including upstream processing elements and downstream processing elements. Each processing element includes input/output port pairs for concurrent forward and back propagation. The processing elements can be used for convolution, in which case the input/output port pairs can support the fast and efficient scanning of kernels relative to activations.