Tensor Transpose Engine With Buffer Scheduling for Memory Bandwidth
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing hardware, such as CPUs and GPUs, inefficiently handle the memory access and reshuffling demands of tensor transposition in machine learning, leading to suboptimal cache memory use and increased memory bandwidth requirements, which slows down computational processes and widens the gap between hardware capabilities and sophisticated machine learning tasks.
Innovation Solution
A processor with an inner transpose engine and address scheduler that utilizes tensor buffer units for internal transposition, reading tensor data row-wise from input memory and writing column-wise into buffer units, and then transferring to output memory, optimizing memory access patterns to match available bandwidth.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Productivity
If CPUs and GPUs are used for tensor transposition, then computational processing is achieved, but memory access efficiency deteriorates and cache memory use becomes suboptimal
Solution Approach 1:
The patent segments the tensor transposition operation into distinct phases: reading tensor data row-wise from input memory, writing column-wise into buffer units, and copying to output memory. This segmentation allows each phase to be optimized independently, with buffer units sized to match channel group dimensions, thereby improving memory access efficiency while maintaining computational processing capability
Solution Approach 2:
The patent introduces buffer units as intermediary structures between input and output memory. These buffer units are specifically sized to match channel group dimensions and enable efficient data transfer through row-wise reading and column-wise writing operations, mediating the memory access pattern to reduce cache misses and improve throughput
2Adaptability or versatility
If tensor transposition is performed with existing hardware, then data manipulation is achieved, but memory bandwidth requirements increase
Solution Approach 1:
The patent implements dynamic buffer unit selection based on channel group sizes. The inner transpose engine selects buffer units whose dimensions match the channel group configuration, allowing the system to adapt memory bandwidth usage to the specific data manipulation requirements of each tensor transposition operation, thereby reducing unnecessary bandwidth consumption while maintaining versatility
3Ease of operation
If conventional tensor transposition is used, then basic data reshuffling is achieved, but computational throughput deteriorates
Solution Approach 1:
The patent performs preliminary buffer unit activation and selection before the actual data transfer begins. The inner transpose engine pre-configures the appropriate buffer units based on channel group dimensions, and the address scheduler pre-calculates memory addresses for both input and output tensors. This preliminary action eliminates runtime delays and optimizes the data reshuffling process, significantly improving computational throughput while maintaining ease of operation
Data Source
AI summary
The present invention relates to a processor designed to optimize memory bandwidth utilization for tensor transpositions in machine learning. It features an instruction decoder that decodes tensor transpose instructions for an input tensor, distinguishing between transposing and stationary axes. An inner transpose engine, equipped with multiple tensor buffer units of varying sizes, transposes the tensor by reading data by rows and writing by columns, efficiently managing memory when the buffer is full. Additionally, an address scheduler determines the target memory addresses in the output tensor memory for the tensor data, improving the handling and transformation of tensors in machine learning environments. This processor significantly enhances the efficiency and speed of tensor operations critical to advanced machine learning applications.