Intelligence Processing Unit Tensor Concatenation via Vector Accelerator
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Conventional tensor concatenation methods are inefficient due to the high time cost of data transfer between memory using direct memory access (DMA) circuits, resulting in poor performance and user experience in electronic devices.
Innovation Solution
An intelligence processing unit (IPU) with a DMA circuit and a vector accelerator, including a register circuit, that reads and stores parts of tensors in a targeted row, allowing for efficient concatenation by optimizing data transfer operations.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Productivity
If conventional DMA circuit performs tensor concatenation by reading and writing tensors between memory 110 and memory 120, then the concatenation operation can be completed, but the number of write operations becomes excessively large (12800 write operations), resulting in low efficiency
Solution Approach 1:
The patent segments the tensor concatenation process into two distinct phases: (1) DMA circuit performs initial data transfer from external memory to internal memory, and (2) vector accelerator completes the concatenation operation by reading from internal memory and writing back to external memory. This segmentation allows each component to perform its optimized function, reducing total operation count significantly.
Solution Approach 2:
The patent introduces an internal memory as an intermediary between external memory and vector accelerator. This intermediary memory buffer enables the vector accelerator to perform efficient concatenation operations without requiring the DMA circuit to handle all data transfer operations, thereby reducing the burden on the DMA circuit and improving overall efficiency.
2Reliability
If DMA circuit performs multiple individual write operations for each tensor, then all tensor data can be transferred to memory 120, but the large number of operations (3200 write operations per tensor) increases processing time and reduces system performance
Solution Approach 1:
The patent merges the concatenation operation with the data transfer operation. Instead of performing separate read and write operations for each tensor, the vector accelerator performs concatenation while transferring data from internal memory to external memory in a unified process, reducing the total number of operations required.
Solution Approach 2:
The patent performs preliminary data transfer using DMA circuit before the vector accelerator performs the concatenation operation. This preliminary action prepares the data in internal memory in a format that enables efficient subsequent processing by the vector accelerator, optimizing the overall workflow.
Data Source
AI summary
An intelligence processing unit is coupled to an external memory and includes a memory, a direct memory access (DMA) circuit, and a vector accelerator. The external memory stores a first tensor and a second tensor. The DMA circuit performs the following steps: reading a first part of the first tensor from the external memory; storing the first part of the first tensor in the memory; reading a second part of the second tensor from the external memory; and storing the second part of the second tensor in the memory. The vector accelerator includes a register circuit and performs the following steps: storing P bytes of the first part of the first tensor in a target row of the register circuit; storing Q bytes of the second part of the second tensor in the target row of the register circuit; and writing data of the target row into the memory.