Word-Based Channels Last Ordering for Neural Network Memory
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Current memory management techniques for neural networks, particularly in edge devices, face inefficiencies due to significant data shuffling required between channels-first and channels-last formats, leading to high computational power usage and bandwidth requirements, especially when accessing 8-bit elements across multiple word boundaries.
Innovation Solution
The introduction of a word-based channels last ordering (WBCLO) memory structure, which organizes bytes within a word from the same row and channel of the tensor, allowing for more efficient reading and writing of data at word boundaries, optimizing memory access for both depth-wise and pointwise convolutions in neural networks.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Productivity
If channels-first memory ordering is used, then data access for depth-wise convolutions is efficient, but significant data shuffling is required for channels-last format operations, increasing computational power usage and bandwidth requirements
Solution Approach 1:
The patent segments the tensor data storage at the word level, where each word contains elements from the same spatial position but different channels. This segmentation allows the memory system to serve both channels-first and channels-last access patterns without complete data shuffling, as words can be accessed in different orders depending on the convolution type being performed.
Solution Approach 2:
The patent introduces a new dimension of organization by grouping tensor elements into words based on spatial position rather than channel order. This creates a third dimension of data organization (word-based grouping) that sits above the traditional channel-first/channels-last dichotomy, enabling efficient access to both formats from a single memory layout.
2Productivity
If channels-first memory ordering is used, then depth-wise convolution operations are efficient, but bandwidth requirements increase for channels-last format operations due to data shuffling
Solution Approach 1:
By segmenting tensor data into word-based groups where each word contains elements from the same spatial position across different channels, the patent enables selective access to only the necessary data for each convolution operation type, reducing the total quantity of data that must be transferred across the memory bandwidth.
Solution Approach 2:
The word-based memory ordering creates a universal memory layout that serves multiple functions: it efficiently supports both depth-wise convolutions (channels-first access pattern) and pointwise convolutions (channels-last access pattern) without requiring separate optimized storage formats, thereby reducing overall bandwidth requirements.
3Ease of operation
If traditional memory access patterns are used for 8-bit elements, then memory access across word boundaries is required, but this increases computational overhead and reduces access efficiency
Solution Approach 1:
The patent performs preliminary organization of tensor data into word-based groups during the data loading or preprocessing stage, arranging elements so that spatially-related data falls within the same word boundaries. This preliminary action ensures that subsequent convolution operations can access all required 8-bit elements within single words, eliminating the need for complex cross-word boundary access patterns.
Data Source
AI summary
A memory device having a plurality of groups of words is provided. Each group of the plurality of groups of words are stored in a contiguous section of a memory and each group of the plurality of groups of words includes a plurality of words. The plurality of words each have a plurality of elements with a same set of index values for a first dimension of a tensor and a same set of index vales for a second dimension of a tensor. The plurality of words have a different index value for a third dimension of the tensor.