Tensor Pointer Remapping for Zero-Overhead Neural Convolutions
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Neural Compute Engines face challenges in efficiently supporting new operations required by evolving Neural Network Architectures, such as ShuffleNet and DeepLab, due to the difficulty in predicting future convolution types and the resource-intensive process of tensor re-formatting, which impacts performance and diverts resources from other tasks.
Innovation Solution
Implementing a Neural Compute Engine that uses storage element pointers for inline re-formatting, allowing it to perform Group and Dilated convolutions without performance overhead, by manipulating tensors directly within the engine.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Adaptability or versatility
If external tensor re-formatting is performed to support new convolution operations, then the Neural Compute Engine can execute diverse neural network architectures, but execution overhead increases and resources are diverted from other tasks
Solution Approach 1:
The patent pre-calculates and stores remapping information (storage element pointers) during tensor formation, before the convolution operation executes. This preliminary action enables the Neural Compute Engine to directly use pre-prepared pointer data for Group and Dilated convolutions, eliminating the need for runtime re-formatting and reducing execution overhead to zero.
Solution Approach 2:
The patent introduces storage element pointers as an intermediary data structure that bridges the gap between standard tensor formats and specialized convolution requirements. These pointers act as a mediator layer that enables diverse convolution operations (Group, Dilated, ShuffleNet) without requiring actual data movement or re-formatting, thus maintaining performance while increasing versatility.
2Adaptability or versatility
If tensor re-formatting is performed externally, then new convolution types can be supported, but resources are diverted from other tasks
Solution Approach 1:
The patent enables the Neural Compute Engine to perform self-service by incorporating remapping information directly within the tensor data structure. The engine uses internal storage element pointers to handle Group and Dilated convolutions without requiring external re-formatting resources, allowing resources to remain available for other tasks while maintaining support for evolving neural network architectures.
Solution Approach 2:
The patent creates a universal tensor representation that incorporates storage element pointers capable of supporting multiple convolution types (standard, Group, Dilated, ShuffleNet) within a single data structure. This multi-functional approach eliminates the need for separate re-formatting processes for different convolution types, optimizing resource utilization while maintaining broad architecture support.
3Ease of operation
If data is moved during re-formatting, then convolution operations can be performed, but execution time increases
Solution Approach 1:
The patent extracts the remapping information (storage element pointers) from the actual tensor data, separating the indexing logic from the computation data. This extraction allows the Neural Compute Engine to perform Group and Dilated convolutions by manipulating pointers rather than moving actual data, thereby maintaining ease of operation while preserving execution speed.
Data Source
AI summary
Methods, apparatus, systems, and articles of manufacture are disclosed for performing a machine learning operation using storage element pointers. An example computer readable medium comprises instructions that when executed, cause at least one processor to select, in response to a determination that a machine learning operation is to be performed, create first and second storage element pointers based on a type of machine learning operation to be performed, remap input tensor data of the input tensor based on the first storage element pointer without movement of the input tensor data in memory, cause execution of the machine learning operation with the remapped input tensor data to create intermediate tensor data, remap the intermediate tensor data based on the second storage element pointer without movement of the intermediate tensor data in memory, and provide the remapped intermediate tensor data as an output tensor.