Tensor Reshape Routing Across Tile Memories for Neural Accelerators
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Reshaping tensors in neural network models on hardware accelerators leads to inefficient memory usage due to the need for a single memory space, which can be insufficient for larger tensors, resulting in underutilization or inability to run certain models, and requires significant data movement between tiles, which is not feasible for smaller tensors.
Innovation Solution
Directly route tensor data between memories of computing units within the hardware accelerator, optimizing memory usage by avoiding a single dedicated memory, using a compiler, reshape solver, and constraint-based solver to efficiently manage data movement based on constraints and schedules.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Adaptability or versatility
If a single dedicated memory is used for reshape operations, then data collection and redistribution can be performed, but memory space is underutilized for smaller tensors or insufficient for larger tensors
Solution Approach 1:
The patent segments the tensor data into multiple chunks and distributes them across multiple existing memory units during reshape operations, rather than using a single dedicated memory. This allows efficient utilization of available memory space across the hardware accelerator while supporting both small and large tensor reshape operations.
Solution Approach 2:
The patent makes existing memory units universal by allowing them to serve both their original computational purposes and reshape operation storage purposes. By routing tensor data directly between these multi-functional memory units, the system eliminates the need for dedicated reshape memory while maintaining full reshape capability.
2Ease of operation
If the entire tensor is moved to a single memory for reshape, then redistribution can be performed, but the operation is not feasible for large tensors due to memory capacity constraints
Solution Approach 1:
The patent divides large tensors into manageable chunks that can be distributed across multiple memory units during reshape operations. This segmentation allows the system to perform reshape operations on tensors of any size by utilizing the collective capacity of all memory units rather than being constrained by a single memory's capacity.
Solution Approach 2:
The patent transitions from a single-memory architecture to a multi-memory architecture, adding the dimension of parallel memory access. This allows the system to handle large tensors by distributing data across multiple memory units simultaneously, effectively increasing the available memory capacity for reshape operations.
3Adaptability or versatility
If a single memory is reserved for reshape operations, then reshape can be performed, but memory space is wasted and model diversity is limited
Solution Approach 1:
The patent makes memory units universal by allowing them to dynamically serve both computational workloads and reshape operations. This multi-functionality enables the system to support diverse neural network models with varying memory requirements while maintaining efficient memory utilization across all operations.
Solution Approach 2:
The patent implements dynamic memory allocation where memory units can switch between computational tasks and reshape operations based on real-time needs. This dynamic approach allows the system to adapt to different model types and sizes without being constrained by static memory reservations, thereby supporting greater model diversity.
4Reliability
If tensor data is moved between tiles during reshape, then local computation can be maintained, but significant data movement is required which increases latency
Solution Approach 1:
The patent performs preliminary routing planning to determine the optimal paths for moving tensor chunks between memory units before executing the reshape operation. This preliminary action optimizes the data movement process, reducing the time required while maintaining the ability to perform local computations on reshaped data.
Data Source
AI summary
A method of performing a reshape operation specified in a reshape layer of a neural network model is described. The reshape operation reshapes an input tensor with an input tensor shape to an output tensor with an output tensor shape. The tensor data that has to be reshaped is directly routed between tile memories of the hardware accelerator in an efficient manner. This advantageously optimizes usage of memory space and allows any number and type of neural network models to be run on the hardware accelerator.