Neural Network Processing Unit Memory Folding
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Current neural network processing units (NPUs) are memory-size inefficient and power-hungry, struggling to handle the high throughput required by real-time AI/ML applications due to their inflexibility in adapting to evolving models and inefficient handling of sparse data.
Innovation Solution
The proposed solution involves folding neighboring pixel activations into a single memory bin, allowing local processing units to access all required activations within a 3*3 neighborhood, enabling efficient storage and processing of convolution operations and stride operations without synchronization, thereby reducing power consumption and increasing processing efficiency.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Productivity
If traditional GPU-based architectures are used for neural network processing, then processing power is sufficient for real-time AI/ML applications, but memory size efficiency deteriorates and power consumption increases
Solution Approach 1:
The patent merges multiple activation bins into a single shared memory bin, allowing multiple processing units to access the same memory location. This consolidation reduces total memory size while maintaining processing capability, directly addressing the contradiction between processing throughput and memory size efficiency
Solution Approach 2:
The shared memory bin serves multiple processing units simultaneously, making a single memory structure universal for multiple operations. This multi-functionality reduces overall memory requirements while maintaining the processing power needed for real-time AI/ML applications
2Productivity
If traditional GPU-based architectures are used for neural network processing, then processing power is sufficient for real-time AI/ML applications, but memory size efficiency deteriorates
Solution Approach 1:
The patent merges multiple activation bins into a single shared memory bin, allowing multiple processing units to access the same memory location. This consolidation reduces total memory size while maintaining processing capability, directly addressing the contradiction between processing throughput and memory size efficiency
Solution Approach 2:
The patent introduces a new dimension of memory access by allowing multiple processing units to simultaneously access the same memory bin through different address mappings. This dimensional change enables reduced memory size while maintaining processing throughput
3Productivity
If dedicated ASICs/ASSPs are used for neural network processing, then processing efficiency is improved, but adaptability to evolving machine learning models deteriorates
Solution Approach 1:
The patent implements a dynamic architecture where processing units can be configured for different operations (convolution, pooling, etc.) and memory structures can be reorganized based on the specific machine learning model being executed. This dynamic reconfigurability maintains processing efficiency while adapting to evolving models
4Ease of operation
If memory bins are expanded to store all activation data, then data accessibility is improved, but memory size efficiency deteriorates
Solution Approach 1:
The patent merges multiple activation bins into a single shared memory bin, improving data accessibility through unified access while reducing total memory size through consolidation
Solution Approach 2:
The patent introduces address mapping logic as an intermediary that translates logical addresses from multiple processing units into physical memory locations. This mediator enables efficient data accessibility while maintaining compact memory structure
Data Source
AI summary
In an example, a method is disclosed of folding each group of neighbor pixels (memory bins) of activations into a same pixel memory bin or a group of 3*3 neighboring pixels memory bins that are all accessible from a middle point processing unit to localize and standardize different convolution operations that are required or other operations such as max pooling or average pooling. The method includes folding together neighboring pixel activations. The method includes storing all the folded activations at the same pixel memory bin so that a local processing unit is able to access all required activations by accessing local memory or 3*3 neighboring pixel memory bins only.