3D Processing Array with Local Memory for Data Flow Optimization
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Current processing systems face inefficiencies in data flow and memory access, particularly in tasks like neural network calculations and large matrix multiplications, where loading model parameters is time-consuming, leading to issues like the 'Batch Size=1 Problem' in data centers.
Innovation Solution
A three-dimensional array of processing elements is arranged with nearest neighbor intercoupling and stacked with DRAM, utilizing through-silicon via connections for high bandwidth and low latency, allowing reconfigurable data flows and local memory access for optimized data exchange and computation.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Loss of time
If data is loaded from external memory for neural network calculations, then model parameters can be accessed, but memory access latency increases and processing efficiency decreases
Solution Approach 1:
The patent implements a nested memory hierarchy where local memory units are integrated within each processing element, which are then stacked in three dimensions. This nested structure allows data to be stored closer to the processing units, reducing access latency while maintaining high capacity through the stacked configuration.
Solution Approach 2:
The patent transitions from traditional two-dimensional memory layouts to a three-dimensional stacked architecture. By adding the vertical dimension through stacking, the system achieves both reduced access paths (improving latency) and increased memory capacity without expanding the footprint, thereby resolving the contradiction between fast access and high productivity.
2Speed
If processing elements are arranged in three-dimensional arrays with nearest neighbor intercoupling, then data flow is optimized and computation speed increases, but device complexity increases
Solution Approach 1:
The system is segmented into multiple processing elements, each with its own local memory unit. This segmentation allows independent operation and optimization of each unit while maintaining simple nearest-neighbor interconnections, avoiding the need for complex global routing infrastructure.
Solution Approach 2:
Each processing element is designed as a universal unit capable of performing the same operations with its local memory. This uniformity simplifies the control logic and interconnection requirements compared to heterogeneous architectures, reducing device complexity while enabling parallel computation for high speed.
3Loss of time
If local memory units are integrated with processing elements, then data exchange is optimized and access latency is reduced, but manufacturing complexity increases
Solution Approach 1:
The patent merges the processing elements and memory units into integrated stacks using three-dimensional integration techniques. This combining approach allows the benefits of local memory (reduced latency) to be achieved while using established semiconductor manufacturing processes for stacking, thereby managing manufacturing complexity.
Data Source
AI summary
An array of processing elements are arranged in a three-dimensional array. Each of the processing elements includes or is coupled to a dedicated memory. The processing elements of the array are intercoupled to their nearest neighbor processing elements. A processing element on a first die may be intercoupled to a first processing element on a second die that is located directly above the processing element, a second processing element on a third die that is located directly below the processing element, and the four adjacent processing elements on the first die. This intercoupling allows data to flow from processing element to processing element in the three directions. These dataflows are reconfigurable so that they may be optimized for the task. The data flows of the array may be configured into one or more loops that periodically recycle data in order to accomplish different parts of a calculation.