Transformer Accelerator Architecture Using Processing-In-Memory
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing accelerator architectures struggle to efficiently accelerate Transformer machine learning models, particularly GPT models, due to their memory-intensive nature and the limitations of compute-in-memory (CIM) architectures, which face challenges with memory density, endurance, and energy consumption.
Innovation Solution
The proposed accelerator architecture incorporates one or more memory devices configured for processing-in-memory (PIM) with a control device to coordinate vector-matrix multiplication operations and other arithmetic tasks. This architecture maximizes local processing and parallelism, reduces latency, and minimizes off-chip data movement through a scheduler scheme and mapping strategy.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Productivity
If compute-in-memory (CIM) architectures are used to accelerate Transformer models, then computation speed is improved, but memory density and endurance deteriorate due to limited SRAM capacity and RRAM wear
Solution Approach 1:
The patent introduces a memory controller as an intermediary between the CPU and DRAM banks, enabling complex VMM operations to be performed directly within the memory subsystem. This mediator coordinates multiple DRAM banks to execute attention mechanisms, thereby improving computation speed while avoiding the reliability issues of CIM architectures by leveraging mature DRAM technology.
Solution Approach 2:
The patent transitions from traditional CPU-based computation to a memory-centric architecture by adding computational capabilities directly within the DRAM subsystem. This dimensional shift moves computation from the processor domain to the memory domain, enabling data to be processed in-place without being transferred to external compute units, thus improving speed while maintaining DRAM reliability.
2Quantity of substance
If high-density on-chip memories such as RRAM are used, then memory density is improved, but energy consumption and endurance worsen due to frequent weight programming
Solution Approach 1:
The patent enables the memory subsystem to perform VMM operations autonomously using its existing storage capacity, eliminating the need for external compute resources. The memory controller coordinates DRAM banks to execute computations directly on stored data, allowing the system to serve its own computational needs without additional energy-intensive weight programming operations.
Solution Approach 2:
The patent makes the DRAM memory subsystem multi-functional by enabling it to perform both storage and computation functions. The same DRAM banks that store model parameters and intermediate results are also used to execute VMM operations, eliminating the need for separate high-density memory arrays like RRAM and their associated energy-consuming programming mechanisms.
3Quantity of substance
If DRAM-based process-in-memory (PIM) technologies are used, then storage capacity and bandwidth are improved, but fabrication complexity increases due to limited metal layers
Solution Approach 1:
The patent divides the computational workload into segments that can be distributed across multiple DRAM banks, each handling specific portions of the VMM operations. The memory controller manages these segmented tasks by coordinating read, write, and compute operations across different banks, thereby achieving high storage capacity and bandwidth while using standard DRAM fabrication processes with limited metal layers.
Solution Approach 2:
The patent implements a dynamic memory controller that adaptively manages computational tasks across DRAM banks based on operational requirements. The controller dynamically allocates resources, adjusts timing, and coordinates operations to optimize performance while working within the constraints of standard DRAM fabrication capabilities, avoiding the need for complex multi-metal layer structures.
Data Source
AI summary
An accelerator architecture is presented for a Transformer machine learning model. The accelerator is comprised of: one or more memory devices, each memory device has a random access memory and is configured for processing in memory, where a key matrix, a value matrix, a query weight matrix, a key weight matrix, and a value weight matrix for an attention mechanism of the Transformer machine learning model are stored in the one or more memory devices; and a control device interfaced with each of the one or more memory devices, the control device is used to coordinate the vector matrix multiplication operations performed on the memory devices, perform other arithmetic and logic operations used in attention blocks that are not suited for the memory devices, and coordinate the updates of the key and value matrices in the one or more memory devices.