HBW NVM Weight Layout for Faster AI Inference Reads
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Conventional memory devices for AI inferencing suffer from insufficient bandwidth, leading to poor performance and high power consumption due to numerous reads/writes, especially in applications requiring hundreds of millions of operations.
Innovation Solution
Implementing a high bandwidth (HBW) NAND flash technology with optimized weight data storage paradigms that reduce the number of read operations and power consumption by storing weights in a page of memory together, allowing sequential retrieval without additional page reads.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Productivity
If conventional memory devices are used for AI inferencing, then storage capacity is sufficient, but bandwidth is insufficient leading to poor performance
Solution Approach 1:
The patent segments weight data into multiple shards distributed across different memory devices. Each shard contains a portion of the weight data for a specific layer or set of layers in the neural network. This segmentation allows parallel access to multiple memory devices simultaneously, effectively increasing the bandwidth available for AI inferencing operations without requiring a single high-capacity memory device.
2Productivity
If conventional memory devices are used for AI inferencing, then storage capacity is sufficient, but power consumption is high due to numerous reads/writes
Solution Approach 1:
The patent performs preliminary actions by pre-processing weight data during system initialization or model loading. Weight data is segmented, shuffled, and distributed across multiple memory devices in advance. During AI inferencing operations, the system simply retrieves pre-organized weight shards without requiring complex real-time data reorganization, thereby reducing the number of read/write operations and lowering power consumption during actual inference.
3Productivity
If weight data is stored traditionally in memory, then storage is simple, but flash access time is slow requiring additional page reads
Solution Approach 1:
The patent merges multiple weight data elements that would traditionally require separate page reads into a single unified shard structure. By organizing weight data so that multiple weight values needed for consecutive compute operations are co-located within the same memory shard, the system can retrieve all required weights in a single read operation, eliminating the need for multiple sequential page reads and significantly reducing access time.
Data Source
AI summary
A method for storing weight data used to compute node values during inferencing operations conducted by a neural network comprises receiving a neural network definition. The neural network definition defines a neural network having a plurality of layers, each having a plurality of nodes. A set of weights used to compute a neural network inferencing result for each neural network node of the plurality of network nodes in the layer is determined. The set of weights determined for the layer in a page of memory can be stored in a high bandwidth non-volatile memory (NVM), such that any weights used to compute the neural network inference result for each neural network node of the plurality of nodes in the layer are stored together in the page of memory for retrieval together. Weights can be stored in different arrays across multiple memory channels.