Tensor Memory Access for Multidimensional Data Transfer Bottlenecks
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Modern processing systems face inefficiencies in memory access speeds, particularly for large data transfers in parallel processing systems, leading to significant computation overhead and performance losses, especially in applications like deep learning, due to complex address calculations and out-of-bounds conditions.
Innovation Solution
A specialized Tensor Memory Access Unit (TMAU) is coupled to each parallel processor core to efficiently transfer large data blocks between global and shared memory, offloading address calculations and handling multidimensional data layouts, enabling asynchronous data transfer and reducing overheads.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Speed
If traditional memory access methods are used in parallel processing systems, then processing cores can perform computations, but memory access speeds are insufficient leading to computation overhead and performance losses
Solution Approach 1:
The patent divides the memory access function into separate specialized units (TMAU) for each processing core. Each TMAU independently handles memory access operations for its associated core, enabling parallel memory access operations without contention and improving overall memory access speed while maintaining computation efficiency.
2Ease of operation
If complex address calculations are performed for multidimensional data structures, then data can be accessed, but significant computation overhead is incurred
Solution Approach 1:
The TMAU acts as an intermediary between the processing core and global memory, taking complex multidimensional data structure access requests from the core and handling the complex address calculations internally. This transfers the computational burden from the general-purpose processing core to the specialized TMAU, reducing computation overhead while maintaining full data access capability.
3Quantity of substance
If data is transferred between global and shared memory using conventional methods, then data transfer is possible, but significant time and energy are consumed
Solution Approach 1:
The TMAU performs preliminary actions by pre-fetching and buffering data from global memory before the processing core needs it for computation. It maintains local buffers and manages data transfer asynchronously, so data is ready when needed without stalling the computation pipeline, thereby reducing data transfer time.
Solution Approach 2:
The TMAU enables continuous data transfer operations by managing multiple data blocks in pipeline fashion. While one block is being transferred, the TMAU can prepare the next block, ensuring continuous useful action without idle periods and minimizing total data transfer time.
4Device complexity
If parallel processor cores handle both computation and memory access operations, then system is simple, but energy consumption increases
Solution Approach 1:
The patent segments the system into computation-specialized cores and memory access-specialized TMAUs. Each core handles computation while its associated TMAU handles memory access operations. This functional segmentation allows each component to operate efficiently in its specialized domain, reducing overall energy consumption despite increased system structural complexity.
Data Source
AI summary
A parallel processing unit comprises a plurality of processors each being coupled to a memory access hardware circuitry. Each memory access hardware circuitry is configured to receive, from the coupled processor, a memory access request specifying a coordinate of a multidimensional data structure, wherein the memory access hardware circuit is one of a plurality of memory access circuitry each coupled to a respective one of the processors; and, in response to the memory access request, translate the coordinate of the multidimensional data structure into plural memory addresses for the multidimensional data structure and using the plural memory addresses, asynchronously transfer at least a portion of the multidimensional data structure for processing by at least the coupled processor. The memory locations may be in the shared memory of the coupled processor and/or an external memory.