Neural Network Weight Reordering for Memory Bandwidth Reduction
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Neural network processing on electronic devices faces challenges due to the large size of weight values, leading to increased power consumption, processing burden, and memory bandwidth usage, particularly in lower power portable devices where resources are constrained.
Innovation Solution
Determining the order of neural network processing and providing weight values in that order to the processor, reducing the need for random memory access and optimizing memory transactions, which can be achieved through reordering and compression of weight values.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If weight values are stored in main memory for neural network processing, then neural network execution is enabled, but power consumption and memory bandwidth usage increase
Solution Approach 1:
The patent segments the large weight matrix into multiple smaller blocks that can be processed independently. Each block is loaded from main memory into local cache memory separately, reducing the total memory bandwidth required and allowing for more efficient processing with lower power consumption.
Solution Approach 2:
The patent implements a nested memory hierarchy where weight blocks are organized in a hierarchical structure with main memory containing all weights, cache memory containing frequently accessed weight blocks, and processor registers containing currently active weights. This nested structure enables efficient data access patterns that reduce power consumption.
2Measurement precision
If weight values are loaded from main memory during processing, then accurate computation is achieved, but memory bandwidth usage increases
Solution Approach 1:
The patent performs preliminary actions by pre-loading weight blocks into cache memory before they are needed for processing. The compiler analyzes the neural network computation graph and prefetches weight blocks that will be required in the near future, ensuring data is available in fast memory when needed without increasing overall bandwidth usage.
Solution Approach 2:
The patent ensures continuous useful action by organizing weight blocks and activation data to maximize cache hit rates. Weight blocks are arranged in memory according to the computation pattern, and processing is scheduled to continuously utilize cached data without idle waits for memory access, maintaining computation accuracy while minimizing bandwidth requirements.
3Speed
If all weight values are kept in local cache memory, then processing speed increases, but device complexity and memory requirements increase
Solution Approach 1:
The patent implements dynamic weight block management where the contents of cache memory are dynamically adjusted based on the current computation stage. The system adaptively loads and evicts weight blocks according to the actual execution flow and data access patterns, optimizing processing speed without requiring excessive cache capacity or complex static memory architecture.
Data Source
AI summary
When performing neural network processing, the order in which the neural network processing is to be performed is determined, and the order in which weight values to be used for the neural network processing will be used is determined based on the determined order of the neural network processing. The weight values are then provided to the processor that is to perform the neural network processing in the determined order for the weight values, with the processor, when performing the neural network processing, then using the weight values in the determined order that they are provided to the processor.