Dynamic Neural Network Hardware via Tensor Segmentation
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Current neural network hardware is limited in implementing dynamic neural networks with variable input tensors, as it is optimized for static tensor sizes, leading to inefficiencies in performance and power consumption when handling inputs with unknown or variable dimensions.
Innovation Solution
A method to transform a dynamic neural network into a static network with fixed-size inputs, allowing it to operate on variable-dimensional inputs by splitting the input tensor into overlapping fixed-size portions and using multiple instances of the static network to process these portions, with combination operations to produce the equivalent output.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Productivity
If hardware is configured to handle networks with predefined static sizes for optimal performance, then execution speed and power efficiency are improved, but the ability to handle variable-dimensional inputs is lost
Solution Approach 1:
The input tensor is segmented into multiple fixed-size portions that can be processed by instances of the static network. This segmentation allows the hardware to maintain its optimized fixed-size configuration while handling variable-dimensional inputs through multiple smaller fixed-size processing units working in parallel or sequence.
Solution Approach 2:
The static network is designed to be universal by accepting fixed-size portions of variable inputs and producing consistent output formats. Multiple instances of this universal network can handle different portions of the input tensor, making the system adaptable to various input sizes while maintaining the same processing architecture.
2Use of energy by stationary object
If hardware is configured to handle networks with predefined static sizes, then memory allocation and operation scheduling can be optimized offline, but flexibility in handling dynamic tensor shapes is reduced
Solution Approach 1:
By segmenting the variable input tensor into fixed-size portions, the system can pre-allocate memory and schedule operations for each portion independently during offline optimization. This maintains the efficiency benefits of static configuration while accommodating dynamic input shapes through the segmented processing approach.
Solution Approach 2:
Memory allocation and operation scheduling are performed in advance for fixed-size portions during the offline phase. This preliminary action allows the hardware to be fully optimized for power efficiency and performance before runtime, while the segmented architecture enables handling of variable-dimensional inputs without sacrificing these pre-optimized benefits.
3Productivity
If the input tensor is split into overlapping fixed-size inputs for processing, then the static network can operate efficiently on each portion, but additional combination operations are required to produce the final output
Solution Approach 1:
The input tensor is divided into overlapping fixed-size portions that can be efficiently processed by the static network. The overlap ensures that boundary regions are properly handled, and the segmentation enables parallel processing of multiple portions, improving overall processing efficiency despite the added complexity of combination operations.
Solution Approach 2:
Multiple output portions from the static network instances are merged or combined to produce the final output tensor. This combining operation integrates the results from processing individual fixed-size portions, reconstructing the complete output while maintaining the efficiency benefits of fixed-size processing throughout the pipeline.
Data Source
Figure 1(a)~1(b)
Figure 2(a)~2(d)
Figure 3
AI summary
A method of implementing in hardware a dynamic neural network for operation on an input tensor having a variable dimension, the method comprising: receiving a representation of the dynamic neural network; transforming the representation of the dynamic neural network into a static network adapted to operate on a fixed size input, the static network being adapted to perform operations on the fixed size input which are equivalent to the operations performed by the dynamic neural network on its input tensor; and implementing a plurality of instances of the static network in hardware for operation on an input tensor split into a sequence of overlapping fixed size inputs along its variable dimension, each instance of the static network being arranged to operate on a respective fixed size input of the sequence.