NVM Inference Engine Memory Hierarchy for Low-Latency Prefetch
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Neural networks place significant demands on memory subsystems, particularly for deep neural networks, due to growing model sizes and datasets, leading to energy consumption, time consumption, and thermal challenges that restrict memory systems and technology.
Innovation Solution
A non-volatile memory-based high-capacity neural network inference engine is implemented, utilizing a stacked memory architecture with high-density memory layers and fast buffer segments to prefetch tasks, reducing latency through ping-pong buffering and integrated AI acceleration.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Quantity of substance
If off-chip memory access is used to handle large model sizes and datasets, then memory capacity is increased, but energy consumption increases and access time increases
Solution Approach 1:
The memory system is segmented into multiple tiers: off-chip memory for bulk storage, on-chip memory for active data, and scratchpad memory for frequent access. This segmentation allows data to be distributed across different memory types based on access frequency, reducing the energy cost of accessing large datasets while maintaining high capacity.
Solution Approach 2:
Data is pre-loaded from off-chip memory into on-chip memory and scratchpad memory before processing. This preliminary action reduces the need for frequent off-chip access during inference, thereby lowering energy consumption while maintaining access to large model sizes and datasets.
2Quantity of substance
If off-chip memory access is used to handle large model sizes and datasets, then memory capacity is increased, but access time increases
Solution Approach 1:
The memory system is segmented into multiple tiers: off-chip memory for bulk storage, on-chip memory for active data, and scratchpad memory for frequent access. This segmentation allows data to be distributed across different memory types based on access frequency, reducing the need for slow off-chip access while maintaining high capacity.
Solution Approach 2:
Data is pre-loaded from off-chip memory into on-chip memory and scratchpad memory before processing. This preliminary action reduces the need for frequent off-chip access during inference, thereby lowering energy consumption while maintaining access to large model sizes and datasets.
3Device complexity
If traditional memory systems are used, then thermal management is simplified, but memory technology choices are restricted
Solution Approach 1:
Different memory technologies are assigned to different locations in the memory hierarchy based on their thermal and performance characteristics. For example, scratchpad memory uses high-speed but thermally intensive technology for frequent access, while off-chip memory uses lower-power technology for bulk storage, optimizing both thermal management and technology choices.
Solution Approach 2:
The system dynamically manages thermal conditions by monitoring temperature and adjusting memory access patterns or data placement. This dynamic approach allows the system to adapt to thermal constraints while utilizing diverse memory technologies, balancing thermal management complexity with memory technology versatility.
Data Source
AI summary
A system, method, and computer program product for a neural network inference engine is disclosed. The inference engine system may include a first memory and a processor in communication with the first memory. The processor may be configured to perform operations. The operations the processor is configured to perform may include fetching a first task with said first memory and delivering the first task to the processor for processing the first task. The operations may further include prefetching a second task with the first memory while the processor is processing the first task. The operations may further include the first memory delivering the second task to the processor upon completion of processing the first task. The operations may further include the processor processing the second task.