Tensor Element Redistribution Using Global Linearized Indexing
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing neural network computing systems face inefficiencies in redistributing tensor elements due to the need for extensive hardware instructions and wide tile-to-tile networks when reshaping or changing tiled dimensions, leading to slower data transfers and computations.
Innovation Solution
Implementing a system where each computing unit uses global linearized index values to determine the destination for tensor elements during redistribution, reducing the need for wide data transfers and utilizing narrower tile-to-tile networks by employing loop nests for efficient tensor element management.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Speed
If tensor elements are redistributed using traditional methods with extensive hardware instructions, then data transfer can be completed, but the data transfer speed is slow and computation efficiency is reduced
Solution Approach 1:
The patent changes the addressing parameter from multi-dimensional indices to global linearized indices. This parameter transformation allows tensor elements to be uniquely identified with a single linear index value, simplifying the redistribution logic and enabling faster data transfer without complex hardware instruction sequences
Solution Approach 2:
The patent segments the tensor redistribution process into independent per-compute-tile operations. Each compute tile independently determines destination addresses for its local tensor elements using the global linearized indexing scheme, eliminating the need for centralized control and reducing hardware instruction complexity
2Productivity
If wide tile-to-tile networks are used to redistribute tensor elements, then all tensor elements can be transferred, but the network width increases and transmission efficiency decreases
Solution Approach 1:
The patent transforms the addressing parameter from multi-dimensional indices requiring wide network addresses to global linearized indices that can be represented with fewer bits. This parameter change enables the use of narrower tile-to-tile networks while maintaining the ability to uniquely address all tensor elements, thereby improving transmission efficiency and computation productivity
3Loss of time
If traditional redistribution methods are used during tensor reshaping, then tensor elements can be moved, but extensive data transfer is required and time is lost
Solution Approach 1:
The patent performs preliminary computation of destination addresses using global linearized indices before the actual data transfer. By pre-calculating where each tensor element should go based on its linearized index and the new tensor shape, the system minimizes unnecessary data movement and reduces redistribution time during tensor reshaping operations
Data Source
AI summary
Methods, systems, and apparatus, including an apparatus for redistributing tensor elements among computing units are described. In one aspect, a method includes distributing tensor elements of an N-dimensional tensor among multiple computing units of a computation system. Each computing unit redistributes the subset of tensor elements previously distributed to the computing unit to computing units. Each computing unit accesses redistribution partitioning data that specifies, for each computing unit, the tensor elements that are to be stored by the computing unit after redistributing the tensor elements. For each tensor element previously distributed to the particular computing unit, the computing unit determines a global linearized index value for the tensor element based on a multi-dimensional index for the tensor element. The computing unit determines, using the redistribution partitioning data and the global linearized index value, a destination computing unit and sends the tensor element to the destination computing unit.