Hierarchical Memory Structure for Neural Network Processing
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Current neural network processing methods face inefficiencies due to the complexities of massive layers and data operations, which are exacerbated by the need for distributed/parallel processing and effective memory management in deep learning applications.
Innovation Solution
A device and method utilizing a hierarchical memory structure comprising SRAM, DRAM, and NAND flash memory, where weights for different layers of a neural network are prefetched and stored based on their read latency characteristics to optimize processing time, allowing for adaptive memory configuration and efficient inference operations.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Quantity of substance
If a hierarchical memory structure with multiple memory types is used, then memory capacity and cost-effectiveness are improved, but memory access time increases
Solution Approach 1:
The system performs preliminary actions by prefetching weights from lower-level memories (NAND flash, DRAM) into upper-level memories (SRAM) before they are actually needed for computation. This is achieved through the weight prefetcher that operates in parallel with the processing units, and the compiler that generates prefetch instructions based on the execution schedule. By preparing data in advance, the system eliminates idle waiting time during inference operations.
Solution Approach 2:
The memory system is segmented into multiple hierarchical levels (SRAM, DRAM, NAND flash) with different characteristics. Each level serves specific purposes: SRAM for frequently accessed weights, DRAM for intermediate storage, and NAND flash for bulk storage. This segmentation allows the system to optimize access patterns by directing different types of memory operations to appropriate levels, balancing capacity and access speed.
2Speed
If weights for multiple layers are stored in fast memory, then processing speed is improved, but device complexity and cost increase
Solution Approach 1:
The memory configuration is made dynamic through the compiler's execution schedule generation. The system determines at compile time which weights should be loaded into which memory levels based on the specific neural network model and inference workload. This dynamic allocation allows the same hardware to be optimized for different models without requiring manual reconfiguration, reducing device complexity while maintaining processing speed.
Solution Approach 2:
The system introduces intermediary components including a weight prefetcher, memory controller, and compiler that mediate between the neural network computation requirements and the hierarchical memory structure. These intermediaries manage the complexity of coordinating multiple memory levels by handling prefetching, caching, and allocation automatically, allowing the processing units to focus on computation while the intermediary layer manages memory complexity.
3Productivity
If distributed/parallel processing is implemented, then processing capacity is improved, but coordination overhead and control complexity increase
Solution Approach 1:
The memory subsystem is designed with universal functionality to serve multiple processing units simultaneously. The weight prefetcher and memory controller can service multiple operation units in parallel, and the hierarchical structure allows any processing unit to access any memory level. This universal design reduces coordination overhead by providing a common memory interface that all units share, rather than requiring dedicated memory controllers for each unit.
Data Source
AI summary
According to an embodiment of the present invention, a device for artificial neural network (ANN) may comprise: memories for read/write (R/W) of data related to an ANN model; and at least one operation unit which performs, based on the data, operations for multiple layers included in the ANN model, wherein the memories include at least one memory-subsystem corresponding to a combination of different types of multiple memories, and each operation unit performs R/W of the data through a memory-subsystem associated with the each operation unit among the at least one memory-subsystem.