Memory-Bank Computing to Break the AI Memory Wall
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
The memory bottleneck in high-performance computing systems, particularly in AI systems using transformer models, limits performance due to the lag in memory access speed compared to processor computation speed, creating a constraint known as the 'memory wall'.
Innovation Solution
A memory device with banks of memory cells and a peripheral circuit that includes a control logic to program and perform calculations on first and second data within the memory device, utilizing process units to distribute calculation tasks and minimize data transfer to the processor, thereby improving calculation speed.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Speed
If memory access speed is increased to match processor computation speed, then computational performance is improved, but device complexity and cost increase
Solution Approach 1:
The memory device is divided into multiple banks of memory cells, each capable of independent data storage and processing. This segmentation allows parallel data access and processing operations, increasing memory access speed without requiring a complete system redesign. Each bank can operate independently to serve different computational tasks simultaneously.
Solution Approach 2:
A peripheral circuit is introduced as an intermediary component between the memory banks and the processor. This peripheral circuit includes control logic and process units that facilitate data transfer and computation, acting as a mediator to bridge the speed gap between memory and processor without directly increasing memory access speed to match processor speeds.
2Quantity of substance
If more data is stored in memory to support large transformer models, then computational capability is improved, but data transfer time increases
Solution Approach 1:
Data is pre-loaded into multiple memory banks in advance of computational tasks. The control logic is configured to program first data and second data into different banks before computation begins, enabling rapid access during processing without requiring extensive data transfer during execution.
Solution Approach 2:
Large datasets are segmented and distributed across multiple memory banks rather than stored in a single location. This allows parallel retrieval and processing of data portions, reducing total data transfer time while maintaining access to large quantities of information for transformer model computations.
3Measurement precision
If computation is performed outside memory device, then processing accuracy is maintained, but data transfer requirements increase
Solution Approach 1:
The peripheral circuit with process units serves as an intermediary processing layer between memory and the main processor. This intermediary can perform preliminary computation and data processing tasks, reducing the volume of data that needs to be transferred to the main processor while maintaining processing accuracy through controlled computation in the memory device.
Solution Approach 2:
Computation tasks are extracted and performed within the memory device using the process units before data is sent to the processor. This extraction of computational functions from the processor reduces the data transfer volume by processing data locally in memory, while the processor receives only necessary results or processed data.
Data Source
AI summary
A memory device, a memory system, and a method for data calculation with the memory device are provided. The memory device includes an array of memory cells and a peripheral circuit coupled to the memory cells is provided. The peripheral circuit includes a control logic configured to program first data and second data into different banks of the banks of memory cells, at least one process unit configured to perform calculation based on the first data and the second data, and a data-path bus coupled to the control logic and the at least one process unit to transmit the first date and the second data.