LoRA Memory Architecture Using HBM and Flash for LLM Training
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Conventional non-volatile memory devices, such as NAND, offer insufficient bandwidth for high-performance applications like large language model (LLM) training, while high-bandwidth volatile memory devices like DRAM are costly and computationally intensive.
Innovation Solution
A computing system utilizing high-bandwidth memory (HBM) units for random access and high-bandwidth flash (HBF) units for storage, employing a low-rank adaptation (LoRA) technique to store and adjust low-rank matrices, optimizing computational efficiency and cost-effectiveness.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Quantity of substance
If conventional non-volatile memory devices (NAND) are used, then storage capacity and cost are improved, but bandwidth is insufficient for high-performance applications
Solution Approach 1:
The patent divides the memory system into two distinct segments: HBM (High Bandwidth Memory) for high-speed data access during training operations, and HBF (High Bandwidth Flash) for bulk storage capacity. This segmentation allows each memory type to operate in its optimal performance zone, with HBM providing the necessary bandwidth for training and HBF providing cost-effective storage for weight matrices.
Solution Approach 2:
The patent merges HBM and HBF into a unified memory architecture that leverages the strengths of both memory types. The system combines the high bandwidth characteristics of HBM with the high storage capacity and cost-effectiveness of HBF, creating a hybrid memory system that achieves both high performance and high capacity simultaneously.
2Productivity
If high-bandwidth volatile memory devices (DRAM/HBM) are used, then bandwidth is improved, but cost and computational intensity increase
Solution Approach 1:
The patent applies local quality by assigning different memory types to different functional requirements: HBM is used locally for data requiring high-speed access during training operations, while HBF is used for bulk storage of weight matrices where high bandwidth is not critical. This localized assignment optimizes cost by using expensive high-bandwidth memory only where absolutely necessary.
Solution Approach 2:
The patent implements a copying mechanism where weight matrices are stored in HBF for long-term retention and copied to HBM only when needed for training operations. This approach allows the system to leverage the cost-effectiveness of HBF for storage while utilizing HBM's high bandwidth only during the relatively brief training periods, significantly reducing overall costs.
3Measurement precision
If full-weight matrices are stored and adjusted during training, then model accuracy is improved, but computational resources and time are excessive
Solution Approach 1:
The patent extracts only the essential components needed for training by storing full-weight matrices in HBF for reference but performing training operations using smaller, optimized data structures in HBM. This extraction approach allows the system to maintain model accuracy by keeping full precision weights available while reducing the computational burden during actual training operations.
Solution Approach 2:
The patent implements dynamic memory management where the system adapts its use of HBM and HBF based on training needs. Full-weight matrices are kept in HBF in a static, high-capacity storage mode, while dynamic training operations utilize HBM's high bandwidth for rapid data access and processing. This dynamic allocation optimizes both accuracy and training time.
Data Source
AI summary
A computing system is provided that includes at least one processing unit, at least one high bandwidth memory (HBM) unit, and at least one high bandwidth flash (HBF) unit. The HBM and HBF units are all in electrical communication with the at least one processing unit. The computing system also includes control circuitry that is configured to train a large language model according to a low-rank adaptation (LoRA) technique. The control circuitry is configured to store a full-weight matrix in the at least one HBF unit and to store at least one low-rank matrix in the at least one HBM unit.